Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
34 commits
Select commit Hold shift + click to select a range
a24e55a
review: one held guest connection and the Tick form of the tickers
CMGS Nov 2, 2026
186816e
review: drop comments that restate their function; docs name the gold…
CMGS Nov 2, 2026
9b95ebc
pool: a template digest guard reads the record, not this node's cache
CMGS Nov 2, 2026
0a27e0c
sdk/go: a canceled TLS handshake reports the cancellation
CMGS Nov 2, 2026
91949c1
review: drop seams nothing injects, an unreachable guard and hand-rol…
CMGS Nov 2, 2026
3deadc7
test: pool tests pass as root, fail instead of hang, and check what t…
CMGS Nov 2, 2026
84a99fe
test: one checkpoint-dir writer, one replica node builder, listTenant…
CMGS Nov 2, 2026
f047376
test: the door cap test waits for a full door before its first request
CMGS Nov 2, 2026
b74151b
test: server tests share the started-server and dial helpers
CMGS Nov 2, 2026
16f89e9
test: the serial-retry test counts blank reads instead of racing a timer
CMGS Nov 2, 2026
0498a87
test: egress and store tests assert what their names claim
CMGS Nov 2, 2026
c093eb0
test: s3 and config tests fail where their names say
CMGS Nov 2, 2026
620bb90
test: intercept method cases share one table and the round-trip bench…
CMGS Nov 2, 2026
5d43c8d
test: Run cancel waits for the pid frame to be consumed
CMGS Nov 2, 2026
915201d
review: e2e package comment is one line
CMGS Nov 2, 2026
ff0ef42
sandbox: refuse a non-bytes stdin instead of hanging the run
CMGS Nov 2, 2026
dec2f10
test: run the fixture data frames through the bytes path and assert d…
CMGS Nov 2, 2026
93c06b7
test: observe the node-side close and the client cut instead of sleep…
CMGS Nov 2, 2026
745ebae
docs: state that archive_failover rows hold claim env in plaintext
CMGS Nov 2, 2026
a55c21e
docs: correct egress forward-door, value_env, lock-free read and remo…
CMGS Nov 2, 2026
84fee1a
docs: list wake-path and release status codes, the env total, and a k…
CMGS Nov 2, 2026
1e99117
docs: correct deploy and cluster statements that drifted from the code
CMGS Nov 2, 2026
04a3d81
docs: fix the SDK example and tenant call, state the egress-lane capt…
CMGS Nov 2, 2026
a951a7b
docs: correct browser, silkd, e2b, benchmarks and timeout wording
CMGS Nov 2, 2026
e23389d
exec: stop the stdin task when its exec finishes
CMGS Nov 2, 2026
0c9405f
fs: keep the destination's owner when root overwrites a file
CMGS Nov 2, 2026
aab1f0f
review: correct the READ_CHUNK, kill_group and lane verdict comments
CMGS Nov 2, 2026
958f08e
sysutil: use nix for the pty's read, write and non-blocking flag
CMGS Nov 2, 2026
7a87374
tree: keep tar's stderr as bytes
CMGS Nov 2, 2026
f15f07c
test: make the silkd tests that could not fail discriminate, and gate…
CMGS Nov 2, 2026
09acdec
ci: run silkd's root-only tests with sudo
CMGS Nov 2, 2026
1c118aa
docs: a silkd write keeps an overwritten file's owner under root
CMGS Nov 2, 2026
a5c7e7f
review: an own-line Go comment starts uppercase (#421)
CMGS Nov 2, 2026
4e9a5d3
exec: stop the stdin task when the client leaves mid-output
CMGS Nov 3, 2026
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 4 additions & 0 deletions .github/workflows/silkd.yml
Original file line number Diff line number Diff line change
Expand Up @@ -33,3 +33,7 @@ jobs:
cargo fmt --check
cargo clippy --locked --all-targets -- -D warnings
cargo test --locked

- name: root-only tests
working-directory: silkd
run: sudo -E env "PATH=$PATH" "HOME=$HOME" cargo test --locked --test git_e2e --test fs_e2e --test find_e2e -- --ignored
5 changes: 3 additions & 2 deletions docs/benchmarks.md
Original file line number Diff line number Diff line change
Expand Up @@ -75,8 +75,9 @@ Knobs (environment variables): `WARM`/`WARM_N` (warm-pool depth and burst
size — the burst must stay within the depth, or refill loses the race and
the tail measures clones), `CLONE_N`, `COLD_N`, `RPC_N`, `PULL_MB`,
`PULL_N`, `BURST_N` (concurrent clone-claim burst; 0 skips the stage),
`ADDR`/`TOKEN` (the throwaway daemon's listen address and token; change them
to run beside a live sandboxd).
`ADDR`/`TOKEN` (the throwaway daemon's listen address and token). Run the
harness on a node with no live sandboxd: the daemon reclaims every `sbx-` VM at
start, and the harness removes every `sbx-` VM at exit.

Boot anatomy (where inside the cold tier the milliseconds go — kernel,
initramfs phases, rootfs handoff) has its own harness:
Expand Down
5 changes: 3 additions & 2 deletions docs/browser.md
Original file line number Diff line number Diff line change
Expand Up @@ -62,8 +62,9 @@ branch from.

- Everything from the Ubuntu flavors (exec, files, sessions, git, pty)
plus the running Chromium.
- **Preview URLs are not raw CDP**: the preview proxy rewrites `Host` to
`sandbox-id:port`, which Chrome's DevTools allowlist rejects. Use
- **Preview URLs are not raw CDP**: the preview proxy forwards the browser's
`Host`, which Chrome's DevTools allowlist rejects unless it is localhost or an
IP. Use
`ProxyPort`/`DialPort` for CDP; preview URLs serve human-facing HTTP the
workload chooses to expose (a live-view page, a screenshot server).
- Guest env knobs read by `/usr/local/bin/chromium-cdp`: `CDP_PORT` (default
Expand Down
12 changes: 7 additions & 5 deletions docs/cluster.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@ A cluster is a set of sandboxd nodes joined through a
Gossip carries only placement hints — per-pool warm counts, promoted-template
hashes, available volume names, each node's data-plane address, and a digest
of the cluster-invariant config.
Per-sandbox state never leaves its owning node, so a stale view costs at most
Gossip carries no per-sandbox state, so a stale view costs at most
one extra redirect, never correctness. A single node with no seeds is a valid
mesh of one.

Expand Down Expand Up @@ -63,7 +63,9 @@ probes, and preview forwarding retain the internal `advertise_addr`. See
[TLS termination](deploy.md#tls-for-sdk-clients) for the edge setup.

Node death is honest: a dead node's sandboxes die with it (memory state is
node-local by design). SWIM detects the death and peers stop redirecting to
node-local by design), except
[archived](#archived-claims-across-node-loss) and
[replicated](#running-claims-across-node-loss) claims. SWIM detects the death and peers stop redirecting to
it.

### Volumes and placement
Expand Down Expand Up @@ -564,8 +566,8 @@ one PostgreSQL table that every node reads, so one tenant per end user works:
listener's next retry. Keepalive finds a dead connection in about 25 s and
retries back off to 30 s, so the change lands within about a minute of the
database returning;
- renew, fork and wake ask the database whenever the cache lacks the
tenant, so they never outlive a removal. While the database is down, they
- renew, fork and wake from the archive ask the database whenever the cache
lacks the tenant, so they never outlive a removal. While the database is down, they
answer 503 for a tenant the node has not cached.

### Cluster-invariant config
Expand Down Expand Up @@ -630,10 +632,10 @@ with every claim.
- the same engine root path (cocoon's `root_dir`) on every node; it is not in
the digest, so a mismatch shows only at the first cross-node branch
- same `api_token`, tenant set, `preview_secret`, and `egress_ca` root everywhere
(a mismatch warns and shows in `sandboxd_config_digest_mismatch`)
- with `meta_store`, every node's DSN points at the same database: the tenant
set is not in the gossiped digest, so nodes on different databases serve
diverging tenants without a warning
(a mismatch warns and shows in `sandboxd_config_digest_mismatch`)
- `cluster_key` set if the gossip network is not otherwise trusted
- one mesh sized in the hundreds of nodes, and promoted templates per pool or
tenant rather than per end user (see [scale](#scale))
Expand Down
24 changes: 15 additions & 9 deletions docs/deploy.md

Large diffs are not rendered by default.

2 changes: 1 addition & 1 deletion docs/e2b.md
Original file line number Diff line number Diff line change
Expand Up @@ -86,7 +86,7 @@ sources that cannot be pinned (a `curl | bash` setup script, an unversioned git
dependency, moving release archives), so they wait for a reproducible source.

Every pip dependency is pinned by `constraints.txt` and `server-constraints.txt`
beside the Dockerfile, and bytecode is compiled hash-checked at the end, so a
beside the Dockerfile, and bytecode is compiled hash-based (unchecked) at the end, so a
rebuild of one commit against the same base yields the same layer. That holds
for the OCI and registry exporters with `SOURCE_DATE_EPOCH` and
`rewrite-timestamp=true`, as the image workflow builds; the `docker` exporter
Expand Down
22 changes: 16 additions & 6 deletions docs/egress.md
Original file line number Diff line number Diff line change
Expand Up @@ -84,7 +84,7 @@ lane needs. The
lock is fail-closed — a claim whose NIC cannot be locked is rejected, not handed
out unlocked, and no policy still means a locked NIC (default-deny), never a free
one. It lives in the host root netns and is removed once the VM is gone (a failed
remove keeps an existing lock in place; the next restart retries the remove). A
remove keeps an existing lock in place; each reap tick retries the remove). A
lock that never applied plus a failed remove leaves the VM unguarded until a
later remove succeeds.

Expand Down Expand Up @@ -248,7 +248,9 @@ woken sandbox binds at arm time.
it only for a claim with a [credential](#claim-credentials) for that host; for
every other claim the rule is a plain one, so its tunnels splice and HTTP/2,
WebSocket and pinned clients keep working. An `"inject"` rule cannot carry a
`secret`. Only a pool rule may set `intercept`.
`secret`. Only a pool rule may set `intercept`. A `true` rule does not serve
the forward door: a plain `http://` or absolute-form `https://` request that
only it matches answers 403.
- No policy on a claim ⇒ no egress at all (the proxy is not started).

Each decision is metered as an `egress` usage event, which names the injected
Expand Down Expand Up @@ -278,16 +280,21 @@ gateway key per user, while the guest holds a placeholder.
which hosts receive a secret, and the claim only supplies its value. A
`guest: true` entry never feeds a secret.
- **Precedence.** The claim's entry wins over the node's value for that claim;
an entry set to `""` suppresses the node's value. With neither, the matched
request passes as the guest sent it.
an entry set to `""` suppresses the node's value. A secret with `value_env`
needs that variable set on the node: unset or empty, it stops the daemon at
start and a reload answers 400. A secret only claims supply omits `value_env`;
on a claim with no entry for it, the matched request passes as the guest sent
it.
- **Lifetime.** The env lives in the claim record in `claims.json` (0600
under `data_dir`), survives a restart, hibernate, wake, archive and
unarchive, and goes with the release or reap. A fork child, a checkpoint
branch and a promoted-template clone start with only the env their own claim
sets. Processes captured live in a snapshot (sessions, detached execs,
units) keep the source's guest values in their own environment; a
`guest: false` value never enters the guest, so no snapshot, checkpoint or
store object holds one.
store object holds one. With [`archive_failover`](deploy.md#configuration),
the row of an archived or replicated claim carries the env, `guest: false`
values included, in plaintext in the shared PostgreSQL.
- **Audit.** An injection is recorded as the secret's name; the value is
never logged, listed or returned.

Expand Down Expand Up @@ -355,6 +362,9 @@ guest's next connection; a tunnel the guest keeps open stays as it was.
The placeholder is not a secret: it stays in the guest and in `GET /env`.
The entry rides the claim request or any `/env` write, and rotation is the
same `PATCH`, accepted on a hibernated or archived claim without waking it.
The write does not reach an archived claim's `archive_failover` row: after that
claim's node is lost, an adopter starts from the row, which holds the value
from the archive.

- **Envelope.** The operator still decides which hosts can carry a
credential: every host must be covered by an `intercept` rule (`true` or
Expand Down Expand Up @@ -431,7 +441,7 @@ that fails answers the guest 502 (SOCKS5: host unreachable) and never falls
back to a direct dial. An allowed connection's audit and usage records name
the upstream's `host:port`, never its credentials, and `GET env` blanks the value like every
`guest: false` entry. Cost: a node without `egress_upstream` pays nothing; with
it, each allowed request reads the claim's env under the manager lock.
it, each allowed request reads the claim's env snapshot, without a lock.

`egress_usage_bytes: true` adds one `egress_bytes` usage event per allowed
tunnel or request when it ends, with the payload bytes it moved (`tx` guest to
Expand Down
2 changes: 2 additions & 0 deletions docs/mcp.md
Original file line number Diff line number Diff line change
Expand Up @@ -44,6 +44,8 @@ in the guest, which can hold its reply up to 5 s past the cap.
| `release` | destroy the sandbox and its files; the session forgets the id, so a second release is rejected as unknown |
| `node_info` | warm pools, promoted templates, live claims, drain state, capacity and mesh peers; needs the operator token |

An egress-lane sandbox cannot `fork`, `checkpoint`, `hibernate` or `promote`.

Sandbox handles (and their tokens) are held by the server process for the
session, and released with it: when the MCP client disconnects or the server
is signalled, it destroys every sandbox that session claimed, whatever
Expand Down
28 changes: 17 additions & 11 deletions docs/sandboxd-api.md
Original file line number Diff line number Diff line change
Expand Up @@ -68,7 +68,7 @@ Auth: `Authorization: Bearer <api_token>` (when configured).
cold-booting an image named like the template
- `env` is the claim's own environment, at most 64 entries: each name a shell
identifier of at most 128 bytes, each value at most 8 KiB with no control
characters other than tab, and names plus values at most 64 KiB in total. An entry holds only `value`, `guest` and `inject`; any other
characters other than tab, and names, values and inject fields at most 64 KiB in total. An entry holds only `value`, `guest` and `inject`; any other
member, or `"guest": null`, answers 400, so a mistyped flag never lands a
host-only value in the guest. `guest` (default `true`) delivers the entry into the guest; a
`guest: false` entry never enters it and only feeds the node's egress
Expand Down Expand Up @@ -475,7 +475,9 @@ independent guest processes and sessions continue. See the
## POST /v1/sandboxes/{id}/release

Auth: the sandbox's own token, or the root token for operator cleanup by id.
Destroys the VM. 204 on success, 404 for an unknown id or wrong token.
Destroys the VM. 204 on success, 404 for an unknown id or wrong token, 500 when
the VM survives removal: the claim is already gone and the node retries the
removal.
Releasing an already-gone sandbox is 404 — the SDK treats it as success.

## POST /v1/sandboxes/{id}/hibernate
Expand Down Expand Up @@ -507,7 +509,8 @@ while stored, and waking it starts a fresh lease of the length the claim was gra
[failed](deploy.md#dead-vmms-and-host-restarts) sandbox is cold-booted from
its own disk (memory is lost) and its restart budget resets. 204 on
success, 404 unknown id or wrong token, 409 a failed egress-lane or volume
sandbox, or one whose VM record is gone, 500 when the cold boot itself fails,
sandbox, or one whose VM record is gone, 403 a removed tenant's archive,
429 while the disk reserve is low, 500 when the cold boot or a restore fails,
503 `lease lost; retry` for a replicated claim while its node's lease is overdue.

## POST /v1/sandboxes/{id}/renew
Expand Down Expand Up @@ -582,7 +585,7 @@ the body:

Publishes the sandbox's current state as a node-local template under
(template, this sandbox's net, its size): later claims for that key clone
from it, provision-on-demand — no warm pool unless the node config adds one.
from it, provision-on-demand, with no warm pool.
Re-promoting to the same name replaces the template. A hibernated sandbox is
promoted from its memory image without waking. 200 returns the template's
full key and immutable content identity. On the default local-disk backend a
Expand Down Expand Up @@ -679,8 +682,8 @@ token; 409 egress pool on a node without an egress attachment; 503 the

## Tenants (/v1/tenants)

Auth: root only (tenant tokens get 403); a node without `api_token` answers 400,
since tenants need it. The API is the only way in: `config.json` has no
Auth: root only (tenant tokens get 403); a node without `api_token` answers 400
to a write, since tenants need it. The API is the only way in: `config.json` has no
`tenants` field, and a node starts with no tenants. The set changes at
runtime, with no restart:

Expand Down Expand Up @@ -893,7 +896,7 @@ an SDK caller should set.
Auth: node API token. Root sees every live claim; a tenant sees only its own.
The index is `{"sandboxes": [{id, key, tenant?, deadline, claimed_at?, hibernated,
archived?, from_checkpoint?, claim_ref?, metadata?, on_expire?, cpu_count, mem_total_bytes,
volumes?: [{name, mount?, mode?, attach_only?}], volumes_want?: [{name, mount?, mode?, attach_only?}], restarts?, restarted_at?, failed?}]}` — `cpu_count` and `mem_total_bytes` are the
volumes?: [{name, mount?, mode?, attach_only?}], volumes_want?: [{name, mount?, mode?, attach_only?}], restarts?, restarted_at?, failed?}]}` — `deadline` is `0001-01-01T00:00:00Z` on an archive kept forever; `cpu_count` and `mem_total_bytes` are the
size tier's allocation; `restarts` counts the cold boots that replaced an
exited VMM and `restarted_at` stamps the last one (each loses guest memory and
the instance-metadata document, so a caller re-establishes guest state when
Expand Down Expand Up @@ -1113,8 +1116,9 @@ clock, so a client that keeps a connection warm must close it when idle for
before the 101, reach silkd in order; the Go and Python SDKs send their first
frames that way over plain HTTP, so a proxy in front of a plain-HTTP endpoint
must forward them too (Go's `httputil.ReverseProxy` drops them; Caddy forwards
them). 426 without the upgrade header, 404 unknown sandbox, 409 a failed
sandbox, 502 guest unreachable, 503 `lease lost; retry` for a replicated claim
them). 426 without the upgrade header, 403 a removed tenant's archive, 404
unknown sandbox, 409 a failed sandbox, 429 while the disk reserve is low, 500 a
failed restore, 502 guest unreachable, 503 `lease lost; retry` for a replicated claim
while its node's lease is overdue (see
[running claims](cluster.md#running-claims-across-node-loss)). A relay of a
claim of a replicating pool that is already open when the lease turns overdue
Expand Down Expand Up @@ -1175,7 +1179,8 @@ fork in flight, answers `409` at once; an open passive relay still holds the san
400 a port outside 1-65535; 401 missing bearer token; 404 unknown sandbox or
wrong token; 409 a failed sandbox or a passive relay to a paused one; 426 without the upgrade
header; 502 when the guest cannot be reached on that port — no listener, or a
hibernated sandbox whose restore failed. The node log carries which; 503
hibernated sandbox whose restore failed. The node log carries which; 403 a
removed tenant's archive; 429 while the disk reserve is low; 503
`lease lost; retry` for a replicated claim while its node's lease is overdue.

## POST /v1/sandboxes/{id}/exec
Expand All @@ -1201,7 +1206,8 @@ does not exist are the caller's, not 502s; 401 missing bearer token; 404 unknown
413 when stdout+stderr exceed 8 MiB; 503 with `Retry-After: 1` when the
node's buffered-exec memory is exhausted (64 MiB of output across every
buffered exec in flight; the command is killed); 502 guest unreachable or any other
silkd error; 409 a failed sandbox; 503 `lease lost; retry` for a replicated claim
silkd error; 403 a removed tenant's archive; 409 a failed sandbox; 429 while the
disk reserve is low; 500 a failed restore; 503 `lease lost; retry` for a replicated claim
while its node's lease is overdue; 504 when the command outlives `timeout_seconds`. A client that hangs up first gets no reply and the command is killed. A hibernated sandbox wakes transparently like on the relay.

## GET /v1/sandboxes/{id}/owner
Expand Down
13 changes: 7 additions & 6 deletions docs/sdk-python.md
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,7 @@ synchronous by design (agent frameworks that need async wrap calls in
does). It matches the [Go SDK](sdk.md) guest and data-plane surface, with these
exceptions: the operator pool and tenant verbs (`SetPools`, `PutTenant` and
their siblings), the claim env (`WithEnv`, `Env`/`SetEnv`/`PatchEnv`) and
setting claim `metadata` or `on_expire` remain Go-only, `Template.new` takes no
setting claim `metadata` or `on_expire`, and reading a claim's `net_route` remain Go-only, `Template.new` takes no
`claim_ref`, and file and tar payloads are whole `bytes` values rather than
streams (`read_file`/`pull` return them, `write_file`/`push` take them; the
wire is chunked, the caller's copy is not). Wire fidelity is
Expand All @@ -23,8 +23,8 @@ Python all round-trip in CI.

## A complete example

Claim a sandbox, push a project in, run a build, then freeze the built state
and fan out two independent workers from that exact moment:
Claim a `node:24.04` sandbox (it carries `make`; `rt:24.04` does not), push a
project in, run a build, then freeze the built state and fan out two independent workers from that exact moment:

```python
import io
Expand All @@ -46,7 +46,7 @@ def project_tar() -> bytes:

def main() -> None:
client = Client(os.environ["SANDBOXD_ADDR"], api_token=os.environ["SANDBOXD_TOKEN"])
with client.new("rt:24.04", size="medium", ttl_seconds=600) as sb:
with client.new("node:24.04", size="medium", ttl_seconds=600) as sb:
print(f"claimed {sb.id} on {sb.owner}")

sb.push("/work", project_tar()) # only ingestion path on the no-network lane
Expand Down Expand Up @@ -123,7 +123,8 @@ branched). `sb.template_digest` is the exact content identity when the claim
cloned a promoted template; it is empty for other sources. `Sandbox` is a
context manager; `sb.close()` releases it (releasing one already gone is not
an error — double-release and reap races stay silent).
Volume sandboxes cannot hibernate, fork, checkpoint, or promote. Checkpoint
Volume sandboxes cannot hibernate, fork, checkpoint, or promote. Egress-lane
sandboxes cannot either; each answers 409. Checkpoint
branches do not accept volumes of either mode in this version.

`mount=False` on `client.new` or `template.new` claims the same volumes
Expand Down Expand Up @@ -389,7 +390,7 @@ code = sb.run(["bash", "-c", "make test"],
```

`timeout` on `exec` and `run` is a wall clock over the dial and the run: at
its end the connection is cut, which makes silkd kill the command, and
its end the SDK cuts the connection and kills the reported pid, and
`TimeoutError` is raised.

`exec` returns stdout and raises `ExitError` on a non-zero exit — carrying
Expand Down
Loading
Loading