From 9b01ce80067d5893037e26f80b1909fa28b0917c Mon Sep 17 00:00:00 2001 From: CMGS Date: Sat, 7 Nov 2026 14:45:35 +0800 Subject: [PATCH 1/7] sandboxd: a missing state for host-only env, kept off disk behind host_env_memory_only --- docs/deploy.md | 6 +- docs/egress.md | 13 ++-- docs/sandboxd-api.md | 23 +++++-- docs/sdk.md | 4 ++ docs/security.md | 8 ++- packaging/sandboxd.service | 1 + sandboxd/config/config.go | 2 + sandboxd/outbound/claim_test.go | 17 +++++ sandboxd/pool/claims.go | 11 +++- sandboxd/pool/env.go | 2 +- sandboxd/pool/env_test.go | 112 +++++++++++++++++++++++++++++++- sandboxd/pool/failover.go | 3 + sandboxd/pool/failover_test.go | 30 ++++++++- sandboxd/pool/hibernate.go | 8 +++ sandboxd/pool/pool.go | 5 +- sandboxd/pool/reconcile.go | 1 + sandboxd/pool/view.go | 2 + sandboxd/server/server_test.go | 22 ++++--- sandboxd/types/env.go | 63 ++++++++++++++---- sandboxd/types/env_test.go | 32 +++++++++ sandboxd/types/types.go | 4 +- sdk/go/env_test.go | 4 +- sdk/go/info.go | 2 + sdk/go/info_test.go | 6 +- sdk/go/options.go | 2 + 25 files changed, 335 insertions(+), 48 deletions(-) diff --git a/docs/deploy.md b/docs/deploy.md index ba916d6e4..59bd77492 100644 --- a/docs/deploy.md +++ b/docs/deploy.md @@ -101,7 +101,8 @@ the intersection of the two. | `bridges` / `networks` | unset | egress-lane attachment: a list of host bridge devices, or a list of CNI conflist names. Mutually exclusive; with neither set the node serves only the no-network lane. A Linux bridge holds at most 1024 ports (kernel `BR_MAX_PORTS`), so an N-entry list raises the node's egress ceiling to N×1024 — VMs spread over the list by a stable hash of the VM name, so size it with headroom (the spread is statistical, not exact). `bridges` keeps the raw TAP-on-bridge attachment (taps in the root netns, no per-VM network namespace or CNI plugin execution); `networks` runs the CNI chain per VM. [Guarded egress](egress.md) on the egress lane (an egress-lane pool policy or any `egress_classes` entry) needs `bridges` and rejects a CNI network at load; none-lane pool policies ride the proxy on either | | `volumes` | unset | node-local catalog of operator-managed dataset images: `[ {"name":"imagenet","path":"/srv/datasets/imagenet.img","directio":"off","tenants":["acme"]}, {"name":"scratch-db","path":"/srv/datasets/scratch.img","writable":true} ]`. Names match `^[a-z][a-z0-9_-]{0,19}$` and cannot start with `cocoon-`; paths are absolute; `directio` is `on`, `off`, or `auto` and defaults to `off` for both read-only and writable entries. `tenants` is an optional [access list](sandboxd-api.md#volume-access-lists); a listed name the tenant set does not hold grants nothing and logs a warning at boot (the [tenant API](sandboxd-api.md#tenants-v1tenants) may add it later). `writable` (default `false`) lets a claim request `mode: "rw"` on that entry — see [Dataset volumes](#dataset-volumes). The catalog is intentionally not part of the cluster digest | | `volume_roots` | unset | absolute directories that bound [runtime registration](sandboxd-api.md#put-v1volumesname): the registered path, once cleaned, must equal a root or sit under one, or the PUT returns 400. The check is lexical: `..` is resolved but symlinks are not followed, so a symlink under a root can point outside it. Static `volumes` entries are not checked. A saved registration outside the roots prevents startup. Unregister it before narrowing the roots, or widen the roots to include it. Keep `data_dir` outside every root. Unset or an empty list accepts any absolute path | -| `secrets` | unset | credentials the egress proxy injects by name: `[{"name": "gh", "header": "Authorization", "value_env": "GH_TOKEN"}]`. A pool or egress-class rule references the name (on a claim where both apply, only the pool rule's secret is injected); the value comes from the claim's `guest: false` env, then the node's environment, never this file. Without `value_env` the secret's name is the env to read, and only claims supply it. See [egress](egress.md#claim-env-and-secrets) | +| `secrets` | unset | credentials the egress proxy injects by name: `[{"name": "gh", "header": "Authorization", "value_env": "GH_TOKEN"}]`. A pool or egress-class rule references the name (on a claim where both apply, only the pool rule's secret is injected); the value comes from the claim's `guest: false` env, then the node's environment, never this file. Without `value_env` the secret's name is the env to read, and only claims supply it. See [egress](egress.md#claim-env-and-secrets). Prefer a claim's own `guest: false` entry over `value_env`: a node value is shared by every claim of the cell, long-lived, and stored in the node's environment file | +| `host_env_memory_only` | `false` | keep claims' `guest: false` env values in memory only: `claims.json` and the [archive and replica rows](#shared-archive-registry) store such an entry without its value and mark it `missing`, and a hibernating claim drops its values. After a restart, a wake from hibernate or the archive, or an adoption, the entry stays `missing`, injects nothing and does not override the node's value until the control plane sends it again with `PATCH` ([claim env](sandboxd-api.md#getputpatch-v1sandboxesidenv)). A live move carries the values to its target. Turn it on only once the control plane re-sends missing entries. Restart only; the first start with it on rewrites `claims.json` without the values | | `egress_internal_allow` | unset | CIDR prefixes re-admitted through the egress proxy's SSRF guard, node-wide (every pool and tenant); `prefix:port,port` scopes an entry to those ports, a bare prefix admits every port. Prefer the port form. Prefixes, not a permit-private switch — the guest bridges are themselves ULA/RFC1918. See [egress](egress.md) | | `egress_upstream` | unset | per-claim upstream proxies: `{"claim_env": "EGRESS_UPSTREAM", "allow": ["res.example.com", "10.1.0.0/16"]}`. A claim's `guest: false` entry under `claim_env` (`http://` or `socks5://`, optional `user:pass@`, or `direct`) routes its egress through that upstream, which must be on `allow` (host names, IPs, CIDR prefixes). See [upstream proxies](egress.md#upstream-proxies) | | `egress_conn_rate` | unset | caps the new upstream connections each claim's proxy opens: `{"per_second": 30, "burst": 600}`, a token bucket per claim; a reused keep-alive connection and a connection to an internal IP address (the SSRF guard decides those) cost nothing, while a host name costs a token even when it resolves to one `egress_internal_allow` re-admits. A connection past it gets 429 (SOCKS5: not allowed) and an audit `decision` of `rate_limited`. A claim's bucket starts full at its first connection, and again when a reload changes `per_second` or `burst`. Unset leaves the rate unlimited | @@ -622,6 +623,9 @@ golden fail the same way, it is retired again at most once every 30 s. The systemd unit is [`packaging/sandboxd.service`](https://github.com/cocoonstack/sandbox/blob/main/packaging/sandboxd.service); install it as shipped. Its `KillMode` and `TimeoutStopSec` are explained below. +`LimitCORE=0` keeps a crash from writing sandboxd's memory, which holds every +running claim's host-only env values, to a core file. For the same reason, +run sandboxd hosts with swap off or encrypted. Each record carries its syslog level, so alerting keys on systemd priority: `journalctl -u sandboxd -p err` lists the errors and `-p warning` the diff --git a/docs/egress.md b/docs/egress.md index 744757059..d1b30fdf8 100644 --- a/docs/egress.md +++ b/docs/egress.md @@ -372,20 +372,25 @@ gateway key per user, while the guest holds a placeholder. which hosts receive a secret, and the claim only supplies its value. A `guest: true` entry never feeds a secret. - **Precedence.** The claim's entry wins over the node's value for that claim; - an entry set to `""` suppresses the node's value. A secret with `value_env` + a `missing` entry ([claim env](sandboxd-api.md#getputpatch-v1sandboxesidenv)) + does not. Prefer claim entries for per-user credentials: a node value is + shared by every claim of the cell and long-lived. A secret with `value_env` needs that variable set on the node: unset or empty, it stops the daemon at start and a reload answers 400. A secret only claims supply omits `value_env`; on a claim with no entry for it, the matched request passes as the guest sent it. - **Lifetime.** The env lives in the claim record in `claims.json` (0600 under `data_dir`), survives a restart, hibernate, wake, archive and - unarchive, and goes with the release or reap. A fork child, a checkpoint + unarchive, and goes with the release or reap. With + [`host_env_memory_only`](deploy.md#configuration) its `guest: false` values + live in memory only and come back `missing` after each of those but a + release, until the control plane sends them again. A fork child, a checkpoint branch and a promoted-template clone start with only the env their own claim sets. Processes captured live in a snapshot (sessions, detached execs, units) keep the source's guest values in their own environment; a `guest: false` value never enters the guest, so no snapshot, checkpoint or - store object holds one; the `archive_failover` row is the exception - ([security](security.md#deployment-assumptions)). + store object holds one; the `archive_failover` row is the exception unless + `host_env_memory_only` is on ([security](security.md#deployment-assumptions)). - **Audit.** An injection is recorded as the secret's name; the value is never logged, listed or returned. diff --git a/docs/sandboxd-api.md b/docs/sandboxd-api.md index 47bcaae80..af969a158 100644 --- a/docs/sandboxd-api.md +++ b/docs/sandboxd-api.md @@ -110,8 +110,9 @@ Auth: `Authorization: Bearer ` (when configured). - `env` is the claim's own environment, at most 64 entries: each name a shell identifier of at most 128 bytes, each value at most 8 KiB, a value and an inject `query` or `placeholder` with no control characters other than tab, and names, values and inject fields at most 64 KiB in total. An entry holds only `value`, `guest` and `inject`; any other - member, or `"guest": null`, answers 400, so a mistyped flag never lands a - host-only value in the guest. `guest` (default `true`) delivers the entry into the guest; a + member, `"guest": null`, a `guest: false` entry with an empty value, or + `missing` (the node's own marker) answers 400, so a mistyped flag never lands + a host-only value in the guest and a read sent back never blanks one. `guest` (default `true`) delivers the entry into the guest; a `guest: false` entry never enters it and only feeds the node's egress [secrets](egress.md#claim-env-and-secrets), or, with `inject` (`{"hosts": [...], "header": "...", "placeholder": "..."}` with `placeholder` @@ -1029,12 +1030,14 @@ an SDK caller should set. Auth: node API token. Root sees every live claim; a tenant sees only its own. The index is `{"sandboxes": [{id, key, tenant?, deadline, claimed_at?, hibernated, archived?, from_checkpoint?, claim_ref?, metadata?, on_expire?, cpu_count, mem_total_bytes, -volumes?: [{name, mount?, mode?, attach_only?}], volumes_want?: [{name, mount?, mode?, attach_only?}], restarts?, restarted_at?, failed?}]}` — `deadline` is `0001-01-01T00:00:00Z` on an archive kept forever; `cpu_count` and `mem_total_bytes` are the +volumes?: [{name, mount?, mode?, attach_only?}], volumes_want?: [{name, mount?, mode?, attach_only?}], restarts?, restarted_at?, failed?, env_missing?}]}` — `deadline` is `0001-01-01T00:00:00Z` on an archive kept forever; `cpu_count` and `mem_total_bytes` are the size tier's allocation; `restarts` counts the cold boots that replaced an exited VMM and `restarted_at` stamps the last one (each loses guest memory and the instance-metadata document, so a caller re-establishes guest state when `restarts` moves; the claim's env is written to the guest again); `failed` names why the VMM is down and was not restarted (see [dead VMMs](deploy.md#dead-vmms-and-host-restarts)); +`env_missing` is true while a `guest: false` env entry has no value on the node +([claim env](#getputpatch-v1sandboxesidenv)); `mode` is omitted for `ro`, matching the claim echo; never sandbox tokens, volume host paths, or catalog access lists. `claimed_at` is the first grant (renew and wake move `deadline` only). `?claim_ref=` keeps only the claims recorded under @@ -1101,11 +1104,23 @@ values: ``` An entry with `inject` comes back with its hosts, header or query, body flag and placeholder, value blanked. + +An entry the node holds no value for comes back with `"missing": true`, and +the claim's index row carries `env_missing`. Under +[`host_env_memory_only`](deploy.md#configuration) that happens after a +restart, a wake from hibernate or the archive, and an adoption. A missing entry +injects nothing, does not override the node's value, and routes no upstream; +send it again with `PATCH` to clear it: + +```json +{"env": {"GW_KEY": {"value": "", "guest": false, "missing": true}}} +``` An `inject` host the claim's pool does not intercept, or its tenant class does not allow, and a header the pool rule's own secret already sets on that host, answer 400 and store nothing. -The env is persisted with the claim. A `guest: false` change takes effect on +The env is persisted with the claim, its `guest: false` values only without +`host_env_memory_only`. A `guest: false` change takes effect on the egress proxy's next request at any time, a hibernated or archived claim included, and never touches the guest. A change to the guest entries is written to the guest at once, so it needs a running guest: on a hibernated, diff --git a/docs/sdk.md b/docs/sdk.md index d42cab45c..131d746ef 100644 --- a/docs/sdk.md +++ b/docs/sdk.md @@ -315,6 +315,10 @@ the claim's owner with the client's API token, not the sandbox token. so a host-only value is never resent. A host-only change applies to the next egress request at any time; a guest change needs a running guest and answers a 409 `*APIError` on a hibernated or archived sandbox. +`Env` reports `Missing` on a host-only entry the node holds no value for, and +`SandboxSummary.EnvMissing` flags the claim; send the entry again with +`PatchEnv`. A host-only entry with an empty value, or one with `Missing` set, +answers 400. ## Renewing diff --git a/docs/security.md b/docs/security.md index 298c01f3b..a2ef44d23 100644 --- a/docs/security.md +++ b/docs/security.md @@ -50,7 +50,11 @@ exposing any part of a deployment beyond a single trusted host. tokens travel as in-memory headers, never guest disk. Secrets come from the host environment or a claim's `guest: false` env, never the config file. That env stays in the node-local claim record, except in the - `archive_failover` row ([assumption 4](#deployment-assumptions)). + `archive_failover` row ([assumption 4](#deployment-assumptions)). With + [`host_env_memory_only`](deploy.md#configuration), `guest: false` values + stay in sandboxd's memory for running claims only: neither file nor row + holds one, and the control plane sends them again after a restart, wake or + adoption. - **Sandbox identity.** Every clone is reseeded (entropy, machine-id), so branches and forks never share an identity with their source. @@ -80,7 +84,7 @@ all of them. held at capture. Restrict store access; use S3 server-side encryption or an encrypted mount where that matters. With `archive_failover`, treat the shared PostgreSQL the same way: its rows hold claim env in plaintext, - `guest: false` values included. + `guest: false` values included unless `host_env_memory_only` is on. 5. **No custom NAT64 prefix routed on a sandboxd host** — the SSRF guard cannot see through an operator-specific translator prefix ([egress](egress.md#deployment-constraints)). diff --git a/packaging/sandboxd.service b/packaging/sandboxd.service index 8b6d3077c..f0f80516a 100644 --- a/packaging/sandboxd.service +++ b/packaging/sandboxd.service @@ -9,6 +9,7 @@ Restart=on-failure RestartSec=5s KillMode=process TimeoutStopSec=27min +LimitCORE=0 Environment=SANDBOXD_LOG_LEVEL=info [Install] diff --git a/sandboxd/config/config.go b/sandboxd/config/config.go index dcbd9e895..0a718b744 100644 --- a/sandboxd/config/config.go +++ b/sandboxd/config/config.go @@ -415,6 +415,8 @@ type Config struct { APIToken string `json:"api_token,omitempty"` MetaStore *MetaStoreConfig `json:"meta_store,omitempty"` Secrets []egress.SecretSpec `json:"secrets,omitempty"` + // HostEnvMemoryOnly keeps host-only env values out of claims.json and the archive rows, and drops them when a claim hibernates. + HostEnvMemoryOnly bool `json:"host_env_memory_only,omitzero"` IdleHibernateSeconds int `json:"idle_hibernate_seconds,omitzero"` ArchiveAfterSeconds int `json:"archive_after_seconds,omitzero"` diff --git a/sandboxd/outbound/claim_test.go b/sandboxd/outbound/claim_test.go index db5406fd9..05cab813d 100644 --- a/sandboxd/outbound/claim_test.go +++ b/sandboxd/outbound/claim_test.go @@ -38,6 +38,7 @@ func TestRouteTakesTheClaimThenClassThenPoolUpstream(t *testing.T) { {"class default", "desk", nil, "http://127.0.0.1:3002"}, {"claim value", "desk", types.Env{"EGRESS_UPSTREAM": {Value: "socks5://127.0.0.1:3003", Guest: hostOnly}}, "socks5://127.0.0.1:3003"}, {"claim direct", "desk", types.Env{"EGRESS_UPSTREAM": {Value: "direct", Guest: hostOnly}}, ""}, + {"claim value missing", "desk", types.Env{"EGRESS_UPSTREAM": {Guest: hostOnly, Missing: true}}, "http://127.0.0.1:3002"}, } { sb := &types.Sandbox{ID: "sb_route", Key: testKey, Tenant: "acme", EgressClass: tt.class} sb.SetEnv(tt.env) @@ -189,6 +190,22 @@ func TestAnInjectShapeRidesIntoTheCredential(t *testing.T) { } } +func TestAMissingEntryInjectsNothingAndLeavesTheNodeSecret(t *testing.T) { + h := newHost(t, &fakeEngine{}, &config.Config{}, nil) + sb := &types.Sandbox{} + sb.SetEnv(types.Env{ + "KEY": {Guest: hostOnly, Inject: &types.EnvInject{Hosts: []string{"api.github.com"}, Header: "X-Api-Key"}, Missing: true}, + "GH_TOKEN": {Guest: hostOnly, Missing: true}, + }) + c := newClaim(h, sb, nil, nil) + if creds := c.Credentials("api.github.com"); len(creds) != 0 { + t.Errorf("credentials %+v, want none for a missing entry", creds) + } + if _, value, ok := c.Header("gh"); !ok || value != "s3cr3t" { + t.Errorf("header %q %v, want the node value", value, ok) + } +} + func TestNewConnectionsPastTheClaimRateAreRefused(t *testing.T) { var dials atomic.Int32 cfg := &config.Config{ diff --git a/sandboxd/pool/claims.go b/sandboxd/pool/claims.go index 4792ea04d..a3948463e 100644 --- a/sandboxd/pool/claims.go +++ b/sandboxd/pool/claims.go @@ -83,8 +83,9 @@ type claimRow struct { // claimStore persists claimed sandboxes across daemon restarts; warm VMs are not persisted. type claimStore struct { - path string - sync bool + path string + sync bool + dropHostEnv bool // writeMu orders commits and guards every row's raw and buf. writeMu sync.Mutex @@ -168,7 +169,11 @@ func (s *claimStore) put(id string, sb *types.Sandbox) { delete(s.rows, id) } else { refs = checkpointRefs(sb) - row := &claimRow{dto: dtoOf(sb), running: holdsVM(sb)} + dto := dtoOf(sb) + if s.dropHostEnv { + dto.Env = dto.Env.DropHostValues() + } + row := &claimRow{dto: dto, running: holdsVM(sb)} if row.running { s.running++ } diff --git a/sandboxd/pool/env.go b/sandboxd/pool/env.go index dd9646d25..f5d965de0 100644 --- a/sandboxd/pool/env.go +++ b/sandboxd/pool/env.go @@ -58,7 +58,7 @@ func (m *Manager) writeEnv(ctx context.Context, id, tenant, verb string, next fu if len(env) == 0 { env = nil } - if err := env.Validate(); err != nil { + if err := env.ValidateStored(); err != nil { m.mu.Unlock() return fmt.Errorf("%w: %w", ErrBadEnv, err) } diff --git a/sandboxd/pool/env_test.go b/sandboxd/pool/env_test.go index 4ed0ad221..f4c5c2f73 100644 --- a/sandboxd/pool/env_test.go +++ b/sandboxd/pool/env_test.go @@ -632,6 +632,112 @@ func TestALostInterceptRefusesOnlyAChangedInject(t *testing.T) { } } +func TestHostEnvMemoryOnlyKeepsValuesOffDiskAndARestartReportsThemMissing(t *testing.T) { + for _, memoryOnly := range []bool{false, true} { + t.Run(fmt.Sprintf("memory_only=%v", memoryOnly), func(t *testing.T) { + eng := newFakeEngine() + dataDir := t.TempDir() + cfg := &config.Config{DataDir: dataDir, APIToken: "root", HostEnvMemoryOnly: memoryOnly} + m := envManagerOf(t, eng, cfg) + sb, err := m.ClaimProvision(t.Context(), testKey, ClaimOptions{Env: types.Env{"gw": {Value: "Bearer a", Guest: hostOnly}, "SEEN": {Value: "yes"}}}) + if err != nil { + t.Fatalf("claim: %v", err) + } + if v, ok := sb.HiddenEnv("gw"); !ok || v != "Bearer a" { + t.Errorf("running claim holds %q %v, want the value in memory", v, ok) + } + claims, err := newClaimStore(dataDir, false).load() + if err != nil { + t.Fatalf("load: %v", err) + } + if gw := claims[sb.ID].Env["gw"]; (gw.Value == "") != memoryOnly || gw.Missing != memoryOnly || claims[sb.ID].Env["SEEN"].Value != "yes" { + t.Errorf("journal gw %+v, want the value kept only without memory_only", gw) + } + + m2 := envManagerOf(t, eng, cfg) + if err = m2.Reconcile(t.Context()); err != nil { + t.Fatalf("Reconcile: %v", err) + } + summary, _ := m2.Sandbox(sb.ID) + listed := m2.Sandboxes("", "", nil) + read, err := m2.Env(sb.ID, "") + if err != nil || summary.EnvMissing != memoryOnly || len(listed) != 1 || listed[0].EnvMissing != memoryOnly || read["gw"].Missing != memoryOnly { + t.Errorf("after a restart: summary %v, list %+v, read %+v %v; want missing reported only with memory_only", summary.EnvMissing, listed, read, err) + } + if err = m2.PatchEnv(t.Context(), sb.ID, types.EnvPatch{"gw": {Value: "Bearer b", Guest: hostOnly}}, ""); err != nil { + t.Fatalf("re-push: %v", err) + } + m2.mu.Lock() + restored := m2.claimed[sb.ID] + m2.mu.Unlock() + summary, _ = m2.Sandbox(sb.ID) + if v, ok := restored.HiddenEnv("gw"); !ok || v != "Bearer b" || summary.EnvMissing { + t.Errorf("after a re-push: %q %v, env_missing %v; want the value back and the flag clear", v, ok, summary.EnvMissing) + } + }) + } +} + +func TestTurningHostEnvMemoryOnlyOnDropsTheValuesAJournalStillHolds(t *testing.T) { + eng := newFakeEngine() + dataDir := t.TempDir() + m := envManagerOf(t, eng, &config.Config{DataDir: dataDir, APIToken: "root"}) + sb, err := m.ClaimProvision(t.Context(), testKey, ClaimOptions{Env: types.Env{"gw": {Value: "Bearer a", Guest: hostOnly}}}) + if err != nil { + t.Fatalf("claim: %v", err) + } + m2 := envManagerOf(t, eng, &config.Config{DataDir: dataDir, APIToken: "root", HostEnvMemoryOnly: true}) + if err = m2.Reconcile(t.Context()); err != nil { + t.Fatalf("Reconcile: %v", err) + } + m2.mu.Lock() + restored := m2.claimed[sb.ID] + m2.mu.Unlock() + claims, err := newClaimStore(dataDir, false).load() + if _, held := restored.HiddenEnv("gw"); held || err != nil || claims[sb.ID].Env["gw"].Value != "" { + t.Errorf("after the first memory-only boot: held %v, journal %+v %v; want the value gone from both", held, claims[sb.ID].Env, err) + } +} + +func TestHostEnvMemoryOnlyDropsValuesWhenAClaimHibernates(t *testing.T) { + for _, memoryOnly := range []bool{false, true} { + t.Run(fmt.Sprintf("memory_only=%v", memoryOnly), func(t *testing.T) { + cfg := &config.Config{DataDir: t.TempDir(), APIToken: "root", HostEnvMemoryOnly: memoryOnly} + m := envManagerOf(t, newFakeEngine(), cfg) + sb, err := m.ClaimProvision(t.Context(), testKey, ClaimOptions{Env: types.Env{"gw": {Value: "Bearer a", Guest: hostOnly}}}) + if err != nil { + t.Fatalf("claim: %v", err) + } + hibernateClaim(t, m, sb) + _, held := sb.HiddenEnv("gw") + summary, _ := m.Sandbox(sb.ID) + if held == memoryOnly || summary.EnvMissing != memoryOnly { + t.Errorf("hibernated claim holds the value %v, env_missing %v; want it dropped only with memory_only", held, summary.EnvMissing) + } + }) + } +} + +func TestHostEnvMemoryOnlyDropsValuesWhenAnUnconfirmedHibernateIsAdopted(t *testing.T) { + eng := newFakeEngine() + m := envManagerOf(t, eng, &config.Config{DataDir: t.TempDir(), APIToken: "root", HostEnvMemoryOnly: true}) + sb, err := m.ClaimProvision(t.Context(), testKey, ClaimOptions{Env: types.Env{"gw": {Value: "Bearer a", Guest: hostOnly}}}) + if err != nil { + t.Fatalf("claim: %v", err) + } + leaveDanglingIntent(t, m, eng, sb) + eng.mu.Lock() + eng.snapListErr = nil + eng.mu.Unlock() + hibernateClaim(t, m, sb) + m.mu.Lock() + snap := sb.HibernateSnap + m.mu.Unlock() + if _, held := sb.HiddenEnv("gw"); held || snap == "" { + t.Errorf("adopted hibernate %q still holds the value %v, want it dropped", snap, held) + } +} + func injectEnv(value string, hosts ...string) types.Env { return types.Env{"KEY": {Value: value, Guest: hostOnly, Inject: &types.EnvInject{Hosts: hosts, Header: "X-Api-Key"}}} } @@ -649,13 +755,17 @@ func credentialsFor(sb *types.Sandbox, host string) string { } func envManager(t *testing.T, eng *fakeEngine, dataDir string, pools ...config.PoolSpec) *Manager { + t.Helper() + return envManagerOf(t, eng, &config.Config{DataDir: dataDir, APIToken: "root", Pools: pools}) +} + +func envManagerOf(t *testing.T, eng *fakeEngine, cfg *config.Config) *Manager { t.Helper() t.Setenv("GH_TOKEN", "node-gh") secrets := testSecrets(t, egress.SecretSpec{Name: "gh", Header: "Authorization", ValueEnv: "GH_TOKEN"}, egress.SecretSpec{Name: "gw", Header: "Authorization"}, ) - cfg := &config.Config{DataDir: dataDir, APIToken: "root", Pools: pools} m, err := NewManager(t.Context(), cfg, eng, secrets, testBackends(t, cfg)) if err != nil { t.Fatalf("manager: %v", err) diff --git a/sandboxd/pool/failover.go b/sandboxd/pool/failover.go index efbe621a5..6715c9e8b 100644 --- a/sandboxd/pool/failover.go +++ b/sandboxd/pool/failover.go @@ -139,6 +139,9 @@ func (m *Manager) putRow(ctx context.Context, sb *types.Sandbox, ck string, repl m.mu.Unlock() dto.Token, dto.VMName, dto.VsockSocket, dto.HibernateSnap, dto.PendingSnap, dto.ArchivePending, dto.ReplicaCk, dto.ReplicaPending = "", "", "", "", "", "", "", "" dto.ArchiveCk, dto.Deadline = ck, deadline + if m.view.Load().hostEnvMemory { + dto.Env = dto.Env.DropHostValues() + } dto.ReaderSafe = dto.ReaderSafe && !replica claim, err := json.Marshal(&dto) if err != nil { diff --git a/sandboxd/pool/failover_test.go b/sandboxd/pool/failover_test.go index 106a50932..3541a9d8d 100644 --- a/sandboxd/pool/failover_test.go +++ b/sandboxd/pool/failover_test.go @@ -4,6 +4,7 @@ import ( "context" "encoding/json/v2" "errors" + "fmt" "os" "path/filepath" "slices" @@ -1868,6 +1869,31 @@ func TestALookupRacingAWakeAdoptionTakesTheRowOnce(t *testing.T) { }) } +func TestFailoverRowsHoldNoHostOnlyValueWhenMemoryOnly(t *testing.T) { + for _, memoryOnly := range []bool{false, true} { + t.Run(fmt.Sprintf("memory_only=%v", memoryOnly), func(t *testing.T) { + c := newFailoverCluster(t) + c.hostEnvMemory = memoryOnly + a, b := c.pair(t) + sb := archivedWithEnv(t, a) + patchHostOnly(t, a, sb, "new") + if row := rowClaim(t, c, sb).Env["GH_TOKEN"]; (row.Value == "") != memoryOnly || row.Missing != memoryOnly { + t.Errorf("row entry %+v, want the value kept only without memory_only", row) + } + c.expire("a") + mustAdopt(t, b, sb, "adopt") + held, ok := b.byID(sb.ID) + if !ok { + t.Fatal("adopter has no claim") + } + summary, _ := b.Sandbox(sb.ID) + if v, set := held.HiddenEnv("GH_TOKEN"); set != !memoryOnly || summary.EnvMissing != memoryOnly || (set && v != "new") { + t.Errorf("adopter holds %q %v, env_missing %v; want the row's value only without memory_only", v, set, summary.EnvMissing) + } + }) + } +} + type failoverCluster struct { reg *archivestest.Mem moves moves.Registry @@ -1877,6 +1903,8 @@ type failoverCluster struct { nodes []*Manager egressPol *egress.Policy noArchives bool + + hostEnvMemory bool } func newFailoverCluster(t *testing.T) *failoverCluster { @@ -1951,7 +1979,7 @@ func (c *failoverCluster) nodeOn(t *testing.T, eng *fakeEngine, name, dataDir st spec.Egress = c.egressPol cfg := &config.Config{ APIToken: "root", DataDir: dataDir, CheckpointDir: c.ckDir, Pools: []config.PoolSpec{spec, {PoolKey: plainKey, Warm: 1}}, - Mesh: &config.MeshConfig{NodeID: name}, Volumes: c.volumes, + Mesh: &config.MeshConfig{NodeID: name}, Volumes: c.volumes, HostEnvMemoryOnly: c.hostEnvMemory, } if c.egressPol != nil { cfg.EgressCA = writeTestEgressCA(t) diff --git a/sandboxd/pool/hibernate.go b/sandboxd/pool/hibernate.go index e01fd42db..112ed7a2d 100644 --- a/sandboxd/pool/hibernate.go +++ b/sandboxd/pool/hibernate.go @@ -313,6 +313,7 @@ func (m *Manager) commitTransition(ctx context.Context, sb *types.Sandbox, snap, } // Under m.mu, so a racing release's event lands after this one and a shutdown cannot keep the record without it if snap != "" { + m.forgetHostEnvLocked(sb) m.recordHibernate(ctx, sb) } else { m.recordUsage(ctx, usage.Event{Event: "wake", ID: sb.ID, VMName: sb.VMName, Tenant: sb.Tenant, Deadline: sb.Deadline}) @@ -355,6 +356,7 @@ func (m *Manager) settlePendingSnap(ctx context.Context, sb *types.Sandbox) (ado if live { if exists { sb.HibernateSnap = pending + m.forgetHostEnvLocked(sb) m.recordHibernate(ctx, sb) } sb.PendingSnap = "" @@ -375,6 +377,12 @@ func (m *Manager) settlePendingSnap(ctx context.Context, sb *types.Sandbox) (ado return exists, nil } +func (m *Manager) forgetHostEnvLocked(sb *types.Sandbox) { + if m.view.Load().hostEnvMemory { + sb.SetEnv(sb.Env.DropHostValues()) + } +} + func (m *Manager) recordHibernate(ctx context.Context, sb *types.Sandbox) { m.counters.hibernates.Add(1) m.recordUsage(ctx, usage.Event{Event: "hibernate", ID: sb.ID, VMName: sb.VMName, Tenant: sb.Tenant}) diff --git a/sandboxd/pool/pool.go b/sandboxd/pool/pool.go index 431448479..5c946ec72 100644 --- a/sandboxd/pool/pool.go +++ b/sandboxd/pool/pool.go @@ -217,6 +217,8 @@ type SandboxSummary struct { Failed string `json:"failed,omitempty"` ClaimedAt time.Time `json:"claimed_at,omitzero"` Deadline time.Time `json:"deadline"` + // EnvMissing reports a host-only env entry whose value the node no longer holds. + EnvMissing bool `json:"env_missing,omitzero"` } // PoolInfo is the ops view of one pool. @@ -438,6 +440,7 @@ func NewManager(ctx context.Context, cfg *config.Config, eng Engine, secrets *eg refillKick: make(chan struct{}, 1), healSem: make(chan struct{}, maxConcurrentHeals), } + m.store.dropHostEnv = cfg.HostEnvMemoryOnly m.seedVolumes(cfg.Volumes) if err := m.loadCordon(); err != nil { return nil, err @@ -733,7 +736,7 @@ func summarize(sb *types.Sandbox) SandboxSummary { ID: sb.ID, Key: sb.Key, Tenant: sb.Tenant, Deadline: sb.Deadline, ClaimedAt: sb.ClaimedAt, Hibernated: sb.HibernateSnap != "", Archived: sb.ArchiveCk != "", FromCheckpoint: sb.FromCheckpoint, ClaimRef: sb.ClaimRef, Metadata: sb.Metadata, OnExpire: sb.OnExpire, - Restarts: sb.Restarts, RestartedAt: sb.RestartedAt, Failed: sb.Failed, + Restarts: sb.Restarts, RestartedAt: sb.RestartedAt, Failed: sb.Failed, EnvMissing: sb.Env.HasMissing(), CPUCount: spec.CPU, MemTotalBytes: spec.MemoryBytes, Volumes: slices.Clone(sb.Volumes), VolumesWant: slices.Clone(sb.VolumesWant), } diff --git a/sandboxd/pool/reconcile.go b/sandboxd/pool/reconcile.go index d65cb5dcf..c0a74a48b 100644 --- a/sandboxd/pool/reconcile.go +++ b/sandboxd/pool/reconcile.go @@ -42,6 +42,7 @@ func (m *Manager) Reconcile(ctx context.Context) error { m.mu.Lock() for _, sb := range claims { sb.SetEnv(sb.Env) + m.forgetHostEnvLocked(sb) rec, ok := live[sb.VMName] if !ok || rec.State != vmStateRunning { sb.SaveSnap = "" diff --git a/sandboxd/pool/view.go b/sandboxd/pool/view.go index 0d37e1a17..b1e401640 100644 --- a/sandboxd/pool/view.go +++ b/sandboxd/pool/view.go @@ -23,6 +23,7 @@ type configView struct { memoryAnon int64 memoryPSI float64 noBalloon bool + hostEnvMemory bool maxClaims int maxRunning int @@ -45,6 +46,7 @@ func newConfigView(cfg *config.Config, secrets *egress.SecretStore) *configView poolReplicate: map[types.PoolKey]time.Duration{}, poolReaders: map[types.PoolKey]bool{}, noBalloon: cfg.NoBalloon, + hostEnvMemory: cfg.HostEnvMemoryOnly, maxClaims: cfg.MaxClaims, maxRunning: cfg.MaxRunning, diff --git a/sandboxd/server/server_test.go b/sandboxd/server/server_test.go index 238b09eb3..76b2ea132 100644 --- a/sandboxd/server/server_test.go +++ b/sandboxd/server/server_test.go @@ -287,10 +287,12 @@ func TestEnvVerbsForRootOrTheTenant(t *testing.T) { t.Errorf("read: %d %s as %q", code, body, mgr.gotTenant) } for body, want := range map[string]int{ - `{"vars":{}}`: http.StatusBadRequest, - `{"env":{"A-B":{"value":"x"}}}`: http.StatusBadRequest, - `{"env":{"A":{"value":"a\nb"}}}`: http.StatusBadRequest, - `{"env":{"A":{"value":"a","x":1}}}`: http.StatusBadRequest, + `{"vars":{}}`: http.StatusBadRequest, + `{"env":{"A-B":{"value":"x"}}}`: http.StatusBadRequest, + `{"env":{"A":{"value":"a\nb"}}}`: http.StatusBadRequest, + `{"env":{"A":{"value":"a","x":1}}}`: http.StatusBadRequest, + `{"env":{"H":{"value":"","guest":false}}}`: http.StatusBadRequest, + `{"env":{"H":{"value":"h","guest":false,"missing":true}}}`: http.StatusBadRequest, } { if code, _ := call(t, http.MethodPut, "root", body); code != want { t.Errorf("%s: %d, want %d", body, code, want) @@ -328,11 +330,13 @@ func TestEnvPatchVerb(t *testing.T) { t.Errorf("PatchEnv calls %q, want %q", mgr.envPatched, want) } for body, want := range map[string]int{ - `{"vars":{}}`: http.StatusBadRequest, - `{"env":{"A-B":null}}`: http.StatusBadRequest, - `{"env":{"A":{"value":"a\nb"}}}`: http.StatusBadRequest, - `{"env":{"A":{"value":"a","x":1}}}`: http.StatusBadRequest, - `{"env":{"A":{"value":"s","guest":null}}}`: http.StatusBadRequest, + `{"vars":{}}`: http.StatusBadRequest, + `{"env":{"A-B":null}}`: http.StatusBadRequest, + `{"env":{"A":{"value":"a\nb"}}}`: http.StatusBadRequest, + `{"env":{"A":{"value":"a","x":1}}}`: http.StatusBadRequest, + `{"env":{"A":{"value":"s","guest":null}}}`: http.StatusBadRequest, + `{"env":{"H":{"value":"","guest":false}}}`: http.StatusBadRequest, + `{"env":{"H":{"value":"h","guest":false,"missing":true}}}`: http.StatusBadRequest, } { if code := call(t, "root", body); code != want { t.Errorf("%s: %d, want %d", body, code, want) diff --git a/sandboxd/types/env.go b/sandboxd/types/env.go index 115da278d..837caebce 100644 --- a/sandboxd/types/env.go +++ b/sandboxd/types/env.go @@ -98,6 +98,8 @@ type EnvVar struct { Guest *bool `json:"guest,omitempty"` // Inject, on a host-only entry, makes the egress proxy send Value to the named hosts. Inject *EnvInject `json:"inject,omitempty"` + // Missing marks a host-only entry whose value the node no longer holds; only the node sets it. + Missing bool `json:"missing,omitzero"` } // InGuest reports whether the entry is delivered into the guest. @@ -106,14 +108,15 @@ func (v EnvVar) InGuest() bool { return v.Guest == nil || *v.Guest } // UnmarshalJSON rejects unknown members and a null guest, so a mistyped host-only flag never falls back to the guest. func (v *EnvVar) UnmarshalJSON(b []byte) error { var raw struct { - Value string `json:"value"` - Guest jsontext.Value `json:"guest,omitzero"` - Inject *EnvInject `json:"inject,omitempty"` + Value string `json:"value"` + Guest jsontext.Value `json:"guest,omitzero"` + Inject *EnvInject `json:"inject,omitempty"` + Missing bool `json:"missing,omitzero"` } if err := json.Unmarshal(b, &raw, json.RejectUnknownMembers(true)); err != nil { return fmt.Errorf("env entry: %w", err) } - *v = EnvVar{Value: raw.Value, Inject: raw.Inject} + *v = EnvVar{Value: raw.Value, Inject: raw.Inject, Missing: raw.Missing} if raw.Guest == nil { return nil } @@ -126,14 +129,27 @@ func (v *EnvVar) UnmarshalJSON(b []byte) error { } func (v EnvVar) same(other EnvVar) bool { - return v.Value == other.Value && v.InGuest() == other.InGuest() && v.Inject.equal(other.Inject) + return v.Value == other.Value && v.InGuest() == other.InGuest() && v.Missing == other.Missing && v.Inject.equal(other.Inject) } // Env is a claim's own environment: guest entries reach the guest, the others stay host-side for egress secrets. type Env map[string]EnvVar -// Validate enforces the entry count, the name grammar, a bounded value a header and an env file both carry, and a bounded total. +// Validate checks a caller's env: ValidateStored's rules, a value on every host-only entry, and no missing entry. func (e Env) Validate() error { + for name, v := range e { + switch { + case v.Missing: + return fmt.Errorf("env %s: missing is set by the node", name) + case !v.InGuest() && v.Value == "": + return fmt.Errorf("env %s: a host-only entry needs a value", name) + } + } + return e.ValidateStored() +} + +// ValidateStored enforces the entry count, the name grammar, a bounded value a header and an env file both carry, and a bounded total. +func (e Env) ValidateStored() error { if len(e) > maxEnvVars { return fmt.Errorf("env must contain at most %d entries, got %d", maxEnvVars, len(e)) } @@ -210,16 +226,24 @@ func (e Env) SameGuest(other Env) bool { return maps.EqualFunc(e.guest(), other.guest(), EnvVar.same) } -// Redacted copies e with every host-only value dropped, so a read names them without serving them. -func (e Env) Redacted() Env { - out := maps.Clone(e) - for name, v := range out { - if !v.InGuest() { - v.Value = "" - out[name] = v +// HasMissing reports whether any host-only entry lacks its value. +func (e Env) HasMissing() bool { + for _, v := range e { + if v.Missing { + return true } } - return out + return false +} + +// Redacted copies e with every host-only value dropped, so a read names them without serving them. +func (e Env) Redacted() Env { + return e.dropHostValues(false) +} + +// DropHostValues copies e with every host-only value dropped and marked missing. +func (e Env) DropHostValues() Env { + return e.dropHostValues(true) } // GuestFile renders the guest entries as a systemd EnvironmentFile, sorted by name. @@ -231,6 +255,17 @@ func (e Env) GuestFile() []byte { return b } +func (e Env) dropHostValues(missing bool) Env { + out := maps.Clone(e) + for name, v := range out { + if !v.InGuest() { + v.Value, v.Missing = "", v.Missing || missing + out[name] = v + } + } + return out +} + func (e Env) guest() Env { out := Env{} for name, v := range e { diff --git a/sandboxd/types/env_test.go b/sandboxd/types/env_test.go index 6a85dd423..cb35357c5 100644 --- a/sandboxd/types/env_test.go +++ b/sandboxd/types/env_test.go @@ -33,6 +33,8 @@ func TestEnvValidate(t *testing.T) { {"over the bound", Env{"A": {Value: strings.Repeat("a", maxEnvValueBytes+1)}}, "at most"}, {"total at the bound", full, ""}, {"total over the bound", over, "total at most"}, + {"host-only without a value", Env{"TOKEN": {Guest: new(false)}}, "needs a value"}, + {"missing from a caller", Env{"TOKEN": {Value: "t", Guest: new(false), Missing: true}}, "set by the node"}, } for _, tt := range tests { t.Run(tt.name, func(t *testing.T) { @@ -219,6 +221,36 @@ func TestEnvSplitsGuestAndHostOnlyEntries(t *testing.T) { } } +func TestDroppedHostValuesAreMissingAndUnpublished(t *testing.T) { + inj := &EnvInject{Hosts: []string{"a.test"}, Header: "X-Key"} + env := Env{"G": {Value: "g"}, "H": {Value: "h", Guest: new(false), Inject: inj}} + dropped := env.DropHostValues() + if h := dropped["H"]; h.Value != "" || !h.Missing || h.Inject != inj || dropped["G"].Value != "g" || env["H"].Value != "h" || env.HasMissing() { + t.Errorf("DropHostValues = %v (source %v), want H missing with its inject kept, G and the source intact", dropped, env) + } + if !dropped.HasMissing() || !dropped.Redacted()["H"].Missing || dropped.Equal(env) { + t.Error("a dropped entry must stay missing through Redacted and differ from the held one") + } + if err := dropped.ValidateStored(); err != nil { + t.Errorf("ValidateStored = %v, want the node's missing entry admitted", err) + } + if err := dropped.Validate(); err == nil { + t.Error("Validate admitted a missing entry from a caller") + } + sb := &Sandbox{} + sb.SetEnv(dropped) + if _, ok := sb.HiddenEnv("H"); ok { + t.Error("a missing entry served a hidden value") + } + for name := range sb.Injections("a.test") { + t.Errorf("a missing entry %s injects", name) + } + var decoded EnvVar + if err := json.Unmarshal([]byte(`{"value":"","guest":false,"missing":true}`), &decoded); err != nil || !decoded.Missing { + t.Errorf("decode = %+v %v, want missing", decoded, err) + } +} + func errMatches(err error, want string) bool { if want == "" { return err == nil diff --git a/sandboxd/types/types.go b/sandboxd/types/types.go index eecc59469..c1e8a8828 100644 --- a/sandboxd/types/types.go +++ b/sandboxd/types/types.go @@ -383,7 +383,7 @@ func (s *Sandbox) SetEnv(e Env) { s.Env = e view := &envView{env: e} for name, v := range e { - if v.Inject != nil && !v.InGuest() { + if v.Inject != nil && !v.InGuest() && !v.Missing { view.injects = append(view.injects, namedVar{name, v}) } } @@ -394,7 +394,7 @@ func (s *Sandbox) SetEnv(e Env) { // HiddenEnv returns a host-only entry's value as last published. func (s *Sandbox) HiddenEnv(name string) (string, bool) { if view := s.live.Load(); view != nil { - if v, ok := view.env[name]; ok && !v.InGuest() { + if v, ok := view.env[name]; ok && !v.InGuest() && !v.Missing { return v.Value, true } } diff --git a/sdk/go/env_test.go b/sdk/go/env_test.go index 191f402fe..b7a5884fd 100644 --- a/sdk/go/env_test.go +++ b/sdk/go/env_test.go @@ -54,7 +54,7 @@ func TestEnvVerbsUseTheAPITokenOnTheOwner(t *testing.T) { body, _ := io.ReadAll(r.Body) got <- request{r.Method, r.Header.Get("Authorization"), string(body)} if r.Method == http.MethodGet { - _, _ = io.WriteString(w, `{"env":{"MODE":{"value":"on"},"GW_KEY":{"value":"","guest":false}}}`) + _, _ = io.WriteString(w, `{"env":{"MODE":{"value":"on"},"GW_KEY":{"value":"","guest":false},"LOST":{"value":"","guest":false,"missing":true}}}`) return } w.WriteHeader(http.StatusNoContent) @@ -67,7 +67,7 @@ func TestEnvVerbsUseTheAPITokenOnTheOwner(t *testing.T) { if err != nil { t.Fatalf("Env: %v", err) } - if want := (map[string]EnvVar{"MODE": {Value: "on"}, "GW_KEY": {Guest: new(false)}}); !reflect.DeepEqual(env, want) { + if want := (map[string]EnvVar{"MODE": {Value: "on"}, "GW_KEY": {Guest: new(false)}, "LOST": {Guest: new(false), Missing: true}}); !reflect.DeepEqual(env, want) { t.Errorf("Env %v, want %v", env, want) } if r := <-got; r != (request{http.MethodGet, "Bearer api", ""}) { diff --git a/sdk/go/info.go b/sdk/go/info.go index f04b4a903..5fed90618 100644 --- a/sdk/go/info.go +++ b/sdk/go/info.go @@ -81,6 +81,8 @@ type SandboxSummary struct { RestartedAt time.Time `json:"restarted_at,omitzero"` // Failed names why the VMM is down and was not restarted; empty means serviceable. Failed string `json:"failed,omitempty"` + // EnvMissing reports a host-only env entry whose value the node no longer holds; re-push it with PatchEnv. + EnvMissing bool `json:"env_missing,omitzero"` } type sandboxListResponse struct { diff --git a/sdk/go/info_test.go b/sdk/go/info_test.go index d0ea754b4..522c2e5cb 100644 --- a/sdk/go/info_test.go +++ b/sdk/go/info_test.go @@ -42,7 +42,7 @@ func TestInfoReportsCapacityState(t *testing.T) { func TestSandboxesDecodeMetadataAndResources(t *testing.T) { ts := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) { _ = json.MarshalWrite(w, map[string]any{"sandboxes": []any{map[string]any{ - "id": "sb_1", "metadata": map[string]string{"team": "a"}, "cpu_count": 2, "mem_total_bytes": 1 << 30, "on_expire": "archive", + "id": "sb_1", "metadata": map[string]string{"team": "a"}, "cpu_count": 2, "mem_total_bytes": 1 << 30, "on_expire": "archive", "env_missing": true, }}}) })) t.Cleanup(ts.Close) @@ -51,7 +51,7 @@ func TestSandboxesDecodeMetadataAndResources(t *testing.T) { if err != nil { t.Fatalf("Sandboxes: %v", err) } - if len(list) != 1 || list[0].Metadata["team"] != "a" || list[0].CPUCount != 2 || list[0].MemTotalBytes != 1<<30 || list[0].OnExpire != "archive" { - t.Errorf("summaries %+v, want metadata team=a, 2 CPUs, 1 GiB", list) + if len(list) != 1 || list[0].Metadata["team"] != "a" || list[0].CPUCount != 2 || list[0].MemTotalBytes != 1<<30 || list[0].OnExpire != "archive" || !list[0].EnvMissing { + t.Errorf("summaries %+v, want metadata team=a, 2 CPUs, 1 GiB and env_missing", list) } } diff --git a/sdk/go/options.go b/sdk/go/options.go index aac6db590..f65b6ebbb 100644 --- a/sdk/go/options.go +++ b/sdk/go/options.go @@ -60,6 +60,8 @@ type EnvVar struct { Value string `json:"value"` Guest *bool `json:"guest,omitempty"` Inject *EnvInject `json:"inject,omitempty"` + // Missing, on a read, marks a host-only entry whose value the node no longer holds; a write that sets it is refused. + Missing bool `json:"missing,omitzero"` } // Option configures a New claim. From 04599d6e524a38cdb4dbccb7b112b016e72cf1b5 Mon Sep 17 00:00:00 2001 From: CMGS Date: Sat, 7 Nov 2026 14:59:27 +0800 Subject: [PATCH 2/7] sandboxd: a missing host-only entry blocks the node's secret and passes the upstream check --- sandboxd/outbound/claim.go | 2 +- sandboxd/outbound/claim_test.go | 6 +++--- sandboxd/outbound/view.go | 2 +- sandboxd/pool/env_test.go | 19 ++++++++++--------- sandboxd/pool/failover_test.go | 4 ++-- sandboxd/pool/upstream_test.go | 1 + sandboxd/types/env_test.go | 4 ++-- sandboxd/types/types.go | 2 +- 8 files changed, 21 insertions(+), 19 deletions(-) diff --git a/sandboxd/outbound/claim.go b/sandboxd/outbound/claim.go index c4e23474a..ab67cd799 100644 --- a/sandboxd/outbound/claim.go +++ b/sandboxd/outbound/claim.go @@ -84,7 +84,7 @@ func (c *claim) Route() string { if view.upstreamEnv == "" { return "" } - if v, set := c.sb.HiddenEnv(view.upstreamEnv); set { + if v, set := c.sb.HiddenEnv(view.upstreamEnv); set && v != "" { if v == upstreamDirect { return "" } diff --git a/sandboxd/outbound/claim_test.go b/sandboxd/outbound/claim_test.go index 05cab813d..41d728fe7 100644 --- a/sandboxd/outbound/claim_test.go +++ b/sandboxd/outbound/claim_test.go @@ -190,7 +190,7 @@ func TestAnInjectShapeRidesIntoTheCredential(t *testing.T) { } } -func TestAMissingEntryInjectsNothingAndLeavesTheNodeSecret(t *testing.T) { +func TestAMissingEntryInjectsNothingAndBlocksTheNodeSecret(t *testing.T) { h := newHost(t, &fakeEngine{}, &config.Config{}, nil) sb := &types.Sandbox{} sb.SetEnv(types.Env{ @@ -201,8 +201,8 @@ func TestAMissingEntryInjectsNothingAndLeavesTheNodeSecret(t *testing.T) { if creds := c.Credentials("api.github.com"); len(creds) != 0 { t.Errorf("credentials %+v, want none for a missing entry", creds) } - if _, value, ok := c.Header("gh"); !ok || value != "s3cr3t" { - t.Errorf("header %q %v, want the node value", value, ok) + if _, value, ok := c.Header("gh"); !ok || value != "" { + t.Errorf("header %q %v, want the node value blocked", value, ok) } } diff --git a/sandboxd/outbound/view.go b/sandboxd/outbound/view.go index fc4c6ede9..89650384c 100644 --- a/sandboxd/outbound/view.go +++ b/sandboxd/outbound/view.go @@ -159,7 +159,7 @@ func (v *View) CheckEnv(env types.Env) error { } e, ok := env[name] switch { - case !ok: + case !ok || e.Missing: return nil case e.InGuest(): return fmt.Errorf("%s must be guest: false, it carries the upstream's credentials", name) diff --git a/sandboxd/pool/env_test.go b/sandboxd/pool/env_test.go index f4c5c2f73..957576239 100644 --- a/sandboxd/pool/env_test.go +++ b/sandboxd/pool/env_test.go @@ -643,8 +643,8 @@ func TestHostEnvMemoryOnlyKeepsValuesOffDiskAndARestartReportsThemMissing(t *tes if err != nil { t.Fatalf("claim: %v", err) } - if v, ok := sb.HiddenEnv("gw"); !ok || v != "Bearer a" { - t.Errorf("running claim holds %q %v, want the value in memory", v, ok) + if v, _ := sb.HiddenEnv("gw"); v != "Bearer a" { + t.Errorf("running claim holds %q, want the value in memory", v) } claims, err := newClaimStore(dataDir, false).load() if err != nil { @@ -671,8 +671,8 @@ func TestHostEnvMemoryOnlyKeepsValuesOffDiskAndARestartReportsThemMissing(t *tes restored := m2.claimed[sb.ID] m2.mu.Unlock() summary, _ = m2.Sandbox(sb.ID) - if v, ok := restored.HiddenEnv("gw"); !ok || v != "Bearer b" || summary.EnvMissing { - t.Errorf("after a re-push: %q %v, env_missing %v; want the value back and the flag clear", v, ok, summary.EnvMissing) + if v, _ := restored.HiddenEnv("gw"); v != "Bearer b" || summary.EnvMissing { + t.Errorf("after a re-push: %q, env_missing %v; want the value back and the flag clear", v, summary.EnvMissing) } }) } @@ -694,8 +694,8 @@ func TestTurningHostEnvMemoryOnlyOnDropsTheValuesAJournalStillHolds(t *testing.T restored := m2.claimed[sb.ID] m2.mu.Unlock() claims, err := newClaimStore(dataDir, false).load() - if _, held := restored.HiddenEnv("gw"); held || err != nil || claims[sb.ID].Env["gw"].Value != "" { - t.Errorf("after the first memory-only boot: held %v, journal %+v %v; want the value gone from both", held, claims[sb.ID].Env, err) + if v, _ := restored.HiddenEnv("gw"); v != "" || err != nil || claims[sb.ID].Env["gw"].Value != "" { + t.Errorf("after the first memory-only boot: held %q, journal %+v %v; want the value gone from both", v, claims[sb.ID].Env, err) } } @@ -709,7 +709,8 @@ func TestHostEnvMemoryOnlyDropsValuesWhenAClaimHibernates(t *testing.T) { t.Fatalf("claim: %v", err) } hibernateClaim(t, m, sb) - _, held := sb.HiddenEnv("gw") + v, _ := sb.HiddenEnv("gw") + held := v != "" summary, _ := m.Sandbox(sb.ID) if held == memoryOnly || summary.EnvMissing != memoryOnly { t.Errorf("hibernated claim holds the value %v, env_missing %v; want it dropped only with memory_only", held, summary.EnvMissing) @@ -733,8 +734,8 @@ func TestHostEnvMemoryOnlyDropsValuesWhenAnUnconfirmedHibernateIsAdopted(t *test m.mu.Lock() snap := sb.HibernateSnap m.mu.Unlock() - if _, held := sb.HiddenEnv("gw"); held || snap == "" { - t.Errorf("adopted hibernate %q still holds the value %v, want it dropped", snap, held) + if v, _ := sb.HiddenEnv("gw"); v != "" || snap == "" { + t.Errorf("adopted hibernate %q still holds %q, want it dropped", snap, v) } } diff --git a/sandboxd/pool/failover_test.go b/sandboxd/pool/failover_test.go index 3541a9d8d..f7113736d 100644 --- a/sandboxd/pool/failover_test.go +++ b/sandboxd/pool/failover_test.go @@ -1887,8 +1887,8 @@ func TestFailoverRowsHoldNoHostOnlyValueWhenMemoryOnly(t *testing.T) { t.Fatal("adopter has no claim") } summary, _ := b.Sandbox(sb.ID) - if v, set := held.HiddenEnv("GH_TOKEN"); set != !memoryOnly || summary.EnvMissing != memoryOnly || (set && v != "new") { - t.Errorf("adopter holds %q %v, env_missing %v; want the row's value only without memory_only", v, set, summary.EnvMissing) + if v, _ := held.HiddenEnv("GH_TOKEN"); (v == "new") == memoryOnly || summary.EnvMissing != memoryOnly { + t.Errorf("adopter holds %q, env_missing %v; want the row's value only without memory_only", v, summary.EnvMissing) } }) } diff --git a/sandboxd/pool/upstream_test.go b/sandboxd/pool/upstream_test.go index 38c2d401b..dc5440b3d 100644 --- a/sandboxd/pool/upstream_test.go +++ b/sandboxd/pool/upstream_test.go @@ -21,6 +21,7 @@ func TestUpstreamEnvIsAdmittedHostOnlyAndAllowed(t *testing.T) { "guest entry": {types.EnvVar{Value: "http://127.0.0.1:3128"}, false}, "unlisted": {types.EnvVar{Value: "http://10.0.0.1:3128", Guest: hostOnly}, false}, "malformed": {types.EnvVar{Value: "ftp://127.0.0.1:21", Guest: hostOnly}, false}, + "missing": {types.EnvVar{Guest: hostOnly, Missing: true}, true}, } { err := m.checkEnv(types.Env{"EGRESS_UPSTREAM": tt.v}) if tt.ok != (err == nil) || (err != nil && !errors.Is(err, ErrBadEnv)) { diff --git a/sandboxd/types/env_test.go b/sandboxd/types/env_test.go index cb35357c5..1194812f4 100644 --- a/sandboxd/types/env_test.go +++ b/sandboxd/types/env_test.go @@ -239,8 +239,8 @@ func TestDroppedHostValuesAreMissingAndUnpublished(t *testing.T) { } sb := &Sandbox{} sb.SetEnv(dropped) - if _, ok := sb.HiddenEnv("H"); ok { - t.Error("a missing entry served a hidden value") + if v, ok := sb.HiddenEnv("H"); !ok || v != "" { + t.Errorf("HiddenEnv(H) = %q %v, want set and empty, so the entry blocks the node's value", v, ok) } for name := range sb.Injections("a.test") { t.Errorf("a missing entry %s injects", name) diff --git a/sandboxd/types/types.go b/sandboxd/types/types.go index c1e8a8828..e52ba7c0c 100644 --- a/sandboxd/types/types.go +++ b/sandboxd/types/types.go @@ -394,7 +394,7 @@ func (s *Sandbox) SetEnv(e Env) { // HiddenEnv returns a host-only entry's value as last published. func (s *Sandbox) HiddenEnv(name string) (string, bool) { if view := s.live.Load(); view != nil { - if v, ok := view.env[name]; ok && !v.InGuest() && !v.Missing { + if v, ok := view.env[name]; ok && !v.InGuest() { return v.Value, true } } From 6e16e12b566b39d88198e36e0f86b307ed9ab9a6 Mon Sep 17 00:00:00 2001 From: CMGS Date: Sat, 7 Nov 2026 15:00:11 +0800 Subject: [PATCH 3/7] docs: when host-only entries go missing, how to resend them, and the upgrade notes --- docs/deploy.md | 11 +++++++++-- docs/egress.md | 4 ++-- docs/sandboxd-api.md | 24 ++++++++++++++---------- docs/sdk.md | 2 +- docs/security.md | 6 +++--- 5 files changed, 29 insertions(+), 18 deletions(-) diff --git a/docs/deploy.md b/docs/deploy.md index 59bd77492..681fed125 100644 --- a/docs/deploy.md +++ b/docs/deploy.md @@ -45,6 +45,13 @@ every archived claim and delete the checkpoints, then re-promote templates, or point `prefix` at an empty location. Objects of the old layout under a reused prefix stay in the bucket until you delete them. +From v0.1.24, a claim env write refuses a `guest: false` entry with an empty +value, so sending back a `GET` env reply answers 400; send each entry with its +value. An empty value no longer blocks the node's secret for one claim. A node +that ran with `host_env_memory_only` stores entries marked `missing`, which +v0.1.24 cannot read: before going back to it, send those entries again or +release their claims, or the node does not start. + ## Configuration sandboxd reads one JSON file (`-config`, default @@ -102,7 +109,7 @@ the intersection of the two. | `volumes` | unset | node-local catalog of operator-managed dataset images: `[ {"name":"imagenet","path":"/srv/datasets/imagenet.img","directio":"off","tenants":["acme"]}, {"name":"scratch-db","path":"/srv/datasets/scratch.img","writable":true} ]`. Names match `^[a-z][a-z0-9_-]{0,19}$` and cannot start with `cocoon-`; paths are absolute; `directio` is `on`, `off`, or `auto` and defaults to `off` for both read-only and writable entries. `tenants` is an optional [access list](sandboxd-api.md#volume-access-lists); a listed name the tenant set does not hold grants nothing and logs a warning at boot (the [tenant API](sandboxd-api.md#tenants-v1tenants) may add it later). `writable` (default `false`) lets a claim request `mode: "rw"` on that entry — see [Dataset volumes](#dataset-volumes). The catalog is intentionally not part of the cluster digest | | `volume_roots` | unset | absolute directories that bound [runtime registration](sandboxd-api.md#put-v1volumesname): the registered path, once cleaned, must equal a root or sit under one, or the PUT returns 400. The check is lexical: `..` is resolved but symlinks are not followed, so a symlink under a root can point outside it. Static `volumes` entries are not checked. A saved registration outside the roots prevents startup. Unregister it before narrowing the roots, or widen the roots to include it. Keep `data_dir` outside every root. Unset or an empty list accepts any absolute path | | `secrets` | unset | credentials the egress proxy injects by name: `[{"name": "gh", "header": "Authorization", "value_env": "GH_TOKEN"}]`. A pool or egress-class rule references the name (on a claim where both apply, only the pool rule's secret is injected); the value comes from the claim's `guest: false` env, then the node's environment, never this file. Without `value_env` the secret's name is the env to read, and only claims supply it. See [egress](egress.md#claim-env-and-secrets). Prefer a claim's own `guest: false` entry over `value_env`: a node value is shared by every claim of the cell, long-lived, and stored in the node's environment file | -| `host_env_memory_only` | `false` | keep claims' `guest: false` env values in memory only: `claims.json` and the [archive and replica rows](#shared-archive-registry) store such an entry without its value and mark it `missing`, and a hibernating claim drops its values. After a restart, a wake from hibernate or the archive, or an adoption, the entry stays `missing`, injects nothing and does not override the node's value until the control plane sends it again with `PATCH` ([claim env](sandboxd-api.md#getputpatch-v1sandboxesidenv)). A live move carries the values to its target. Turn it on only once the control plane re-sends missing entries. Restart only; the first start with it on rewrites `claims.json` without the values | +| `host_env_memory_only` | `false` | keep claims' `guest: false` env values in memory only: `claims.json` and the [archive and replica rows](#shared-archive-registry) store such an entry without its value and mark it `missing`, and a hibernating claim drops its values. A missing entry injects nothing and blocks the node's value for its secret until the control plane sends it again with `PATCH` ([claim env](sandboxd-api.md#getputpatch-v1sandboxesidenv)). A live move carries the values to its target. Turn it on only when every node of the cell runs v0.1.25 or later with it, and the control plane sends missing entries again. The first start with it on rewrites `claims.json` without the values; an archive row written before keeps its values until the row is next written. Restart only | | `egress_internal_allow` | unset | CIDR prefixes re-admitted through the egress proxy's SSRF guard, node-wide (every pool and tenant); `prefix:port,port` scopes an entry to those ports, a bare prefix admits every port. Prefer the port form. Prefixes, not a permit-private switch — the guest bridges are themselves ULA/RFC1918. See [egress](egress.md) | | `egress_upstream` | unset | per-claim upstream proxies: `{"claim_env": "EGRESS_UPSTREAM", "allow": ["res.example.com", "10.1.0.0/16"]}`. A claim's `guest: false` entry under `claim_env` (`http://` or `socks5://`, optional `user:pass@`, or `direct`) routes its egress through that upstream, which must be on `allow` (host names, IPs, CIDR prefixes). See [upstream proxies](egress.md#upstream-proxies) | | `egress_conn_rate` | unset | caps the new upstream connections each claim's proxy opens: `{"per_second": 30, "burst": 600}`, a token bucket per claim; a reused keep-alive connection and a connection to an internal IP address (the SSRF guard decides those) cost nothing, while a host name costs a token even when it resolves to one `egress_internal_allow` re-admits. A connection past it gets 429 (SOCKS5: not allowed) and an audit `decision` of `rate_limited`. A claim's bucket starts full at its first connection, and again when a reload changes `per_second` or `burst`. Unset leaves the rate unlimited | @@ -139,7 +146,7 @@ the intersection of the two. | `idle_hibernate_seconds` | 0 (off) | node-wide idle policy for unpooled claims (template/checkpoint claims): a none-lane claim is hibernated once it has had no open data-plane connection (relay, buffered exec, preview), no egress request in flight and no [advanced-network](advanced-network.md) packet carried for this long; the clock restarts when the last connection closes, so a long command is never cut short. The next call that reaches the guest wakes it transparently. Per-pool `idle_hibernate_seconds` does the same for that pool's claims; pooled keys ignore the node-wide value, and egress pools reject it because they cannot resume safely. Opt in deliberately: a wake costs latency and the snapshot, so callers with their own idle logic must not pay twice | | `archive_after_seconds` | 0 (off) | tier below hibernation: a hibernated claim idle this long is checkpointed to the store and its local VM dropped, freeing the node entirely; the next call that reaches the guest restores it transparently (a checkpoint restore's latency) with a fresh lease of the length the claim was granted (the server default of 5m when it asked for none); a lease moves only when a client calls [`/renew`](sandboxd-api.md#post-v1sandboxesidrenew). Keep `idle_hibernate_seconds` under the leases your claims use, or a woken sandbox that idles again is destroyed by the reap before it can hibernate, its archive already consumed by the wake; one still in use when its lease ends is destroyed the same way, unless it claimed `on_expire: archive`, which archives at the lease end whatever this setting. A call that wakes a hibernated claim past its lease, before the export, starts that fresh lease too. Requires `idle_hibernate_seconds > 0` and must exceed it. Node-wide for unpooled keys; per-pool overrides for that pool | | `archive_delete_after_seconds` | 0 (keep) | purge an archived claim's store checkpoint this long after it was archived; the claim is then gone for good. On S3, chunks and manifests remain until [offline reclamation](#reclaiming-s3-storage). On archive this retention window replaces the live claim deadline; 0 clears the deadline so the archive is kept forever. An [evacuation](cluster.md#evacuating-a-node) keeps a longer live deadline, and a claim of a pool without an archive tier, whose `on_expire` is not `archive`, keeps its deadline until its first wake. Same node-wide/per-pool split | -| `archive_failover` | false | let another node adopt this node's archived claims once its node lease expires, and the running claims of pools with `replicate_seconds`: each archive and replication round also writes a row (token SHA-256, owner, epoch, checkpoint id, retention deadline, and the claim record with its env, `guest: false` values included, in plaintext) to the shared PostgreSQL, and a `Lookup` that reaches any node takes the claim over there (see [archived claims across node loss](cluster.md#archived-claims-across-node-loss)). Requires `meta_store` with a `cell` (rows are scoped by it, so meshes sharing one database never adopt from each other), a `checkpoint_store` of kind `s3` and a `mesh` with a `cluster_key` (a hand-off presents the claim's token to an address learned from gossip), all checked at config load. The node also renews a lease row every 5 s (see [node lease](cluster.md#node-lease)). With the meta store down, archives, wakes, releases and env writes of archived claims, and env writes and renews of hibernated replicated claims, fail and retry, and replicated claims stop being served once the lease is overdue (see [running claims](cluster.md#running-claims-across-node-loss)); other running claims never touch it. Restart only | +| `archive_failover` | false | let another node adopt this node's archived claims once its node lease expires, and the running claims of pools with `replicate_seconds`: each archive and replication round also writes a row (token SHA-256, owner, epoch, checkpoint id, retention deadline, and the claim record with its env, `guest: false` values included, in plaintext, unless `host_env_memory_only` is on) to the shared PostgreSQL, and a `Lookup` that reaches any node takes the claim over there (see [archived claims across node loss](cluster.md#archived-claims-across-node-loss)). Requires `meta_store` with a `cell` (rows are scoped by it, so meshes sharing one database never adopt from each other), a `checkpoint_store` of kind `s3` and a `mesh` with a `cluster_key` (a hand-off presents the claim's token to an address learned from gossip), all checked at config load. The node also renews a lease row every 5 s (see [node lease](cluster.md#node-lease)). With the meta store down, archives, wakes, releases and env writes of archived claims, and env writes and renews of hibernated replicated claims, fail and retry, and replicated claims stop being served once the lease is overdue (see [running claims](cluster.md#running-claims-across-node-loss)); other running claims never touch it. Restart only | | `mesh` | unset | join a cluster ([Clusters](cluster.md)); unset = single node | | `pools[]` | — | warm pools, keyed by `(template, net, size)`. `warm` defaults to 4 only when both `warm` and `warm_max` are omitted; an explicit `warm: 0` is kept, so `warm: 0` under a `warm_max` starts the pool empty and grows it on demand, and `warm: 0` alone registers the key with no warm VMs (`PUT /v1/pools` never defaults `warm`); `net` is `none` (the default) or `egress`; `size` is a tier, below (default `small`). Retune online without a restart via [`PUT /v1/pools`](sandboxd-api.md#put-v1pools) — omitted pools drain, and their goldens are deleted once they are idle (`warm: 0` keeps the golden). This is the **first-boot seed**: once a node takes a `PUT /v1/pools`, the applied set persists to `/pools.json` and overrides this section on every later boot (a startup log notes it); delete `pools.json` to return to config-owned pools. With a `meta_store` `cell`, the cell's set overrides it instead (see [shared pool set](#shared-pool-set)). Egress stays config-owned either way. A template re-pulled or re-imported under the same ref is picked up within a minute: the pool rebuilds its golden and replaces its warm VMs. See [state ownership](cluster.md#state-ownership) | diff --git a/docs/egress.md b/docs/egress.md index d1b30fdf8..481c01ca6 100644 --- a/docs/egress.md +++ b/docs/egress.md @@ -373,7 +373,7 @@ gateway key per user, while the guest holds a placeholder. `guest: true` entry never feeds a secret. - **Precedence.** The claim's entry wins over the node's value for that claim; a `missing` entry ([claim env](sandboxd-api.md#getputpatch-v1sandboxesidenv)) - does not. Prefer claim entries for per-user credentials: a node value is + blocks the node's value until it is sent again. Prefer claim entries for per-user credentials: a node value is shared by every claim of the cell and long-lived. A secret with `value_env` needs that variable set on the node: unset or empty, it stops the daemon at start and a reload answers 400. A secret only claims supply omits `value_env`; @@ -475,7 +475,7 @@ changed: resend it, and the resend replaces the row. - **Precedence.** The pool rule's own `secret` is set first and wins: a credential naming that header for a host the rule covers is a 400. Among credentials for the same header, the first by entry name whose placeholder - (if any) the request carries wins. An empty value injects nothing. + (if any) the request carries wins. A missing entry injects nothing. - **Echo.** Origins reflect the request URL: a redirect's `Location`, a paging link, an echoed query. On a request that carried a `query` credential, its value, as sent, query- or path-escaped, or with `/` written diff --git a/docs/sandboxd-api.md b/docs/sandboxd-api.md index af969a158..810ce5c35 100644 --- a/docs/sandboxd-api.md +++ b/docs/sandboxd-api.md @@ -1088,13 +1088,13 @@ it reaches; on a cluster, send it to the claim's `owner_addr`. rules; `{"env": {}}` clears it. `PATCH` merges: each entry in the body is set, an entry of `null` removes that name, and every other entry is kept as stored, host-only values included, so a control plane that cannot read those values -back never has to send them again: +back sends them again only when the node reports them missing (below): ```json {"env": {"CB_META_wake": {"value": "w2"}, "OLD": null}} ``` -The claim rules apply to the merged env, so a patch that takes it past 64 +The size rules apply to the merged env, so a patch that takes it past 64 entries or 64 KiB answers 400 and changes nothing. `GET` answers the same shape with every `guest: false` value blanked, so a control plane sees the names without the values: @@ -1104,20 +1104,24 @@ values: ``` An entry with `inject` comes back with its hosts, header or query, body flag and placeholder, value blanked. +An `inject` host the claim's pool does not intercept, or its tenant class does +not allow, and a header the pool rule's own secret already sets on that host, +answer 400 and store nothing. An entry the node holds no value for comes back with `"missing": true`, and the claim's index row carries `env_missing`. Under -[`host_env_memory_only`](deploy.md#configuration) that happens after a -restart, a wake from hibernate or the archive, and an adoption. A missing entry -injects nothing, does not override the node's value, and routes no upstream; -send it again with `PATCH` to clear it: +[`host_env_memory_only`](deploy.md#configuration) that happens when the claim +hibernates, after a restart and after an adoption. A missing entry injects +nothing and blocks the node's value for its secret; a missing upstream entry +falls back to the class or pool upstream. Send it again once the claim runs: +a `PATCH` replaces the entry whole, so repeat `guest` and `inject`: ```json -{"env": {"GW_KEY": {"value": "", "guest": false, "missing": true}}} +{"env": {"GW_KEY": {"value": "Bearer …", "guest": false, "inject": {"hosts": ["api.example.com"], "header": "Authorization"}}}} ``` -An `inject` host the claim's pool does not intercept, or its tenant class does -not allow, and a header the pool rule's own secret already sets on that host, -answer 400 and store nothing. + +A value sent to a hibernated claim stays in that node's memory until the +claim hibernates again. The env is persisted with the claim, its `guest: false` values only without `host_env_memory_only`. A `guest: false` change takes effect on diff --git a/docs/sdk.md b/docs/sdk.md index 131d746ef..a03027bac 100644 --- a/docs/sdk.md +++ b/docs/sdk.md @@ -316,7 +316,7 @@ so a host-only value is never resent. A host-only change applies to the next egress request at any time; a guest change needs a running guest and answers a 409 `*APIError` on a hibernated or archived sandbox. `Env` reports `Missing` on a host-only entry the node holds no value for, and -`SandboxSummary.EnvMissing` flags the claim; send the entry again with +`SandboxSummary.EnvMissing` flags the claim; send the whole entry again with `PatchEnv`. A host-only entry with an empty value, or one with `Missing` set, answers 400. diff --git a/docs/security.md b/docs/security.md index a2ef44d23..9841f08c8 100644 --- a/docs/security.md +++ b/docs/security.md @@ -52,9 +52,9 @@ exposing any part of a deployment beyond a single trusted host. file. That env stays in the node-local claim record, except in the `archive_failover` row ([assumption 4](#deployment-assumptions)). With [`host_env_memory_only`](deploy.md#configuration), `guest: false` values - stay in sandboxd's memory for running claims only: neither file nor row - holds one, and the control plane sends them again after a restart, wake or - adoption. + stay in sandboxd's memory: `claims.json` and the rows written with it on hold + none, and a hibernating claim drops them. The control plane sends them again + after a hibernate, a restart or an adoption. - **Sandbox identity.** Every clone is reseeded (entropy, machine-id), so branches and forks never share an identity with their source. From f5bef4f7409b955f36c975250c1fc845355223b0 Mon Sep 17 00:00:00 2001 From: CMGS Date: Sat, 7 Nov 2026 15:08:18 +0800 Subject: [PATCH 4/7] docs: turn host_env_memory_only off before re-sending entries for a downgrade --- docs/deploy.md | 7 +++++-- 1 file changed, 5 insertions(+), 2 deletions(-) diff --git a/docs/deploy.md b/docs/deploy.md index 681fed125..495f353ad 100644 --- a/docs/deploy.md +++ b/docs/deploy.md @@ -49,8 +49,11 @@ From v0.1.24, a claim env write refuses a `guest: false` entry with an empty value, so sending back a `GET` env reply answers 400; send each entry with its value. An empty value no longer blocks the node's secret for one claim. A node that ran with `host_env_memory_only` stores entries marked `missing`, which -v0.1.24 cannot read: before going back to it, send those entries again or -release their claims, or the node does not start. +v0.1.24 cannot read. Before going back to it, turn the flag off and restart, +then send every missing entry again or release its claim, until no claim's +`env_missing` is true: each write stores the value in `claims.json` and the +claim's row. A `claims.json` that still holds a missing entry stops v0.1.24 +from starting, and a row that holds one cannot be adopted by it. ## Configuration From 854a73f7d9bfc47b6ff4854e1975875bc43a50fd Mon Sep 17 00:00:00 2001 From: CMGS Date: Sat, 7 Nov 2026 15:10:51 +0800 Subject: [PATCH 5/7] docs: wait one replica interval before a downgrade so replica rows lose missing --- docs/deploy.md | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/docs/deploy.md b/docs/deploy.md index 495f353ad..bedafbf5c 100644 --- a/docs/deploy.md +++ b/docs/deploy.md @@ -51,8 +51,9 @@ value. An empty value no longer blocks the node's secret for one claim. A node that ran with `host_env_memory_only` stores entries marked `missing`, which v0.1.24 cannot read. Before going back to it, turn the flag off and restart, then send every missing entry again or release its claim, until no claim's -`env_missing` is true: each write stores the value in `claims.json` and the -claim's row. A `claims.json` that still holds a missing entry stops v0.1.24 +`env_missing` is true, then wait one `replicate_seconds` interval. Each write +stores the value in `claims.json` and in the row of an archived or hibernated +claim; a running replicated claim's row takes it at its next replica. A `claims.json` that still holds a missing entry stops v0.1.24 from starting, and a row that holds one cannot be adopted by it. ## Configuration From cf5a8f51919dd44e6907a4d5c7441e946d899af3 Mon Sep 17 00:00:00 2001 From: CMGS Date: Sat, 7 Nov 2026 15:12:26 +0800 Subject: [PATCH 6/7] docs: bound when a running replica row takes a re-sent value --- docs/deploy.md | 11 +++++++---- 1 file changed, 7 insertions(+), 4 deletions(-) diff --git a/docs/deploy.md b/docs/deploy.md index bedafbf5c..5bb708a46 100644 --- a/docs/deploy.md +++ b/docs/deploy.md @@ -51,10 +51,13 @@ value. An empty value no longer blocks the node's secret for one claim. A node that ran with `host_env_memory_only` stores entries marked `missing`, which v0.1.24 cannot read. Before going back to it, turn the flag off and restart, then send every missing entry again or release its claim, until no claim's -`env_missing` is true, then wait one `replicate_seconds` interval. Each write -stores the value in `claims.json` and in the row of an archived or hibernated -claim; a running replicated claim's row takes it at its next replica. A `claims.json` that still holds a missing entry stops v0.1.24 -from starting, and a row that holds one cannot be adopted by it. +`env_missing` is true. Each write stores the value in `claims.json` and in the +row of an archived or hibernated claim. A running replicated claim's row takes +it only at its next replica, up to two `replicate_seconds` intervals plus one +replica's capture after the restart; to rewrite the row at once, hibernate the +claim and send its entries again. A `claims.json` that still holds a missing +entry stops v0.1.24 from starting, and a row that holds one cannot be adopted +by it. ## Configuration From 46a397a55ae77988b7f15c1890315fc39cb881ee Mon Sep 17 00:00:00 2001 From: CMGS Date: Sat, 7 Nov 2026 15:14:20 +0800 Subject: [PATCH 7/7] docs: rewrite running replica rows by hibernating before a downgrade --- docs/deploy.md | 9 ++++----- 1 file changed, 4 insertions(+), 5 deletions(-) diff --git a/docs/deploy.md b/docs/deploy.md index 5bb708a46..0dc8da21e 100644 --- a/docs/deploy.md +++ b/docs/deploy.md @@ -53,11 +53,10 @@ v0.1.24 cannot read. Before going back to it, turn the flag off and restart, then send every missing entry again or release its claim, until no claim's `env_missing` is true. Each write stores the value in `claims.json` and in the row of an archived or hibernated claim. A running replicated claim's row takes -it only at its next replica, up to two `replicate_seconds` intervals plus one -replica's capture after the restart; to rewrite the row at once, hibernate the -claim and send its entries again. A `claims.json` that still holds a missing -entry stops v0.1.24 from starting, and a row that holds one cannot be adopted -by it. +it only at a later replica, with no fixed bound, so hibernate each such claim +and send its entries again; a write that succeeds has rewritten its row. A +`claims.json` that still holds a missing entry stops v0.1.24 from starting, and +a row that holds one cannot be adopted by it. ## Configuration