diff --git a/README.md b/README.md index e260e7e28..d7cd40baf 100644 --- a/README.md +++ b/README.md @@ -6,6 +6,11 @@ in-guest product daemon, a per-node control plane with warm pools, and SDKs for Go and Python. Warm claims are sub-millisecond; a pool miss clones from a golden snapshot in tens of milliseconds; cold boot is ~0.2–0.4 s on bare metal. +**Documentation: [cocoonstack.github.io/sandbox](https://cocoonstack.github.io/sandbox/)** +(source in [`docs/`](docs/)). + +## Architecture + ``` SDK (Go/Python) sandboxd (per node) guest microVM client.New/new ─ HTTP ─► claim: warm pool / golden clone Cloud Hypervisor @@ -17,120 +22,83 @@ exec/files/… ─ HTTP upgrade (kept) ─► byte relay ─ vsock ─► silkd Cloud Hypervisor serves both network lanes. `net=none` has no NIC and uses vsock-only I/O (hardened default); `net=egress` attaches a bridge/CNI NIC. -**Documentation**: [cocoonstack.github.io/sandbox](https://cocoonstack.github.io/sandbox/) -(deployment, clusters, HTTP API, Go + Python SDK references, the MCP server, -the OpenAI Agents SDK and LangChain adapters, silkd protocol, -performance) — source in -[`docs/`](docs/). +## Quick start -## Layout +On a Linux node with KVM, [cocoon](https://github.com/cocoonstack/cocoon) v0.6.9 +or newer, Cloud Hypervisor and `jq`, as root, from a checkout with Go installed. +Not on a node that already runs sandboxd on the same cocoon root: at start +sandboxd removes every `sbx-` VM it does not own. -- `silkd/` — in-guest product daemon (Rust, tokio): exec with context, - persistent shell sessions, streaming fs, tar-stream tree push/pull, - find/replace, watch (ready-acked), pty, structured git, guest port - relay (`port_forward`), and an LSP broker for flavor-shipped language - servers — newline-JSON frames over vsock 2048, RPCs back to back on one - connection; - baked into the base image -- `sandboxd/` — per-node control plane (Go): warm pools refilled from golden - snapshot exports (online-retunable), claim/release/renew/hibernate/fork/ - promote/checkpoint HTTP API, operator-catalog dataset volumes (static or - registered at runtime, read-only or writable, mounted or attach-only, with - live add and remove), signed preview URLs, - the HTTP-upgrade byte relays to silkd and to any guest port, read-only file - access to hibernated or archived claims from a warm VM of their pool - (opt-in per pool `reader`), claim env with - host-side credential injection and per-claim upstream proxies, runtime - tenants, config reload (SIGHUP), usage + audit journals, /metrics, reap + - restart reconcile with VMM cold restart, memberlist mesh with redirect - placement, a checkpoint store on a shared directory or S3 (sparse, - content-addressed chunks), and an optional shared PostgreSQL meta store - for tenants, pools, node leases, usage events and the cross-node adoption of - archived claims and of running claims a pool replicates (`replicate_seconds`) -- `sdk/go/` — Go SDK: `Connect/New/Lookup`, `Exec/Run`, files, - `Push/Pull`, sessions, `Find/Replace`, `Watch`, git verbs, `OpenPty`, - `Renew/Fork/Hibernate/Promote/Checkpoint`, claim env, `DialPort/ProxyPort/PreviewURL`, - `GetVolumes/SetVolumes/PatchVolumes`, - `StartLsp`, `Spawn/Ps/Kill/Logs/Attach`; `protocol/wire` carries the frame - vocabulary, `sdk/go/silkd` the conn layer, `silkdtest` a test fake -- `sdk/python/` — Python SDK (stdlib-only, sync), matching the Go guest and - data-plane surface for the Python-first agent ecosystem; node pool and - tenant administration and the claim env remain Go-only; round-trips the - shared fixture corpus -- `mcp/` — `sandbox-mcp`, an MCP stdio server exposing the surface as tools - for Claude Code / Cursor / agent frameworks -- `sdk/openai/` — `cocoonstack-sandbox-openai`, a custom sandbox provider - for the OpenAI Agents SDK (Python, over the Python SDK) -- `sdk/langchain/` — `cocoonstack-sandbox-langchain`, a LangChain toolkit - (StructuredTools over the Python SDK, checkpoint branching) -- `protocol/wire/fixtures/` — golden frame corpus; the Rust, Go, and Python - protocol tests all round-trip it, so wire drift fails CI -- `e2e/` — in-process full-stack tests (real pool/engine/relay/SDK, fake - cocoon+guest) plus bare-metal acceptance drivers under `cmd/`: `demo`, - `smoke`, `meshsmoke`, `crossnode`, `coldproof`, `egresssmoke`, - `interceptsmoke`, `sockssmoke`, `volumesmoke`, `lifecycle` (idle→hibernate→archive), - `androidsmoke`, `browsersmoke`, `ringsmoke` (output ring cap, exec as a user), - `portsmoke` (the guest-port relay, with `guestserver` as its guest listener), - `credsmoke` (claim credential injection), - and the `pullbench`/`pushbench`/`rpcbench`/`qaab` perf drivers -- `boot/kernel/` — kernel version pin (`VERSION` + matching tarball `SHA256`, - bump both together) + config fragment (amd64: over `x86_64_defconfig` + - `kvm_guest.config`; arm64: over `defconfig`, then `sandbox-arm64.config`) -- `boot/init/` — `sandbox-init`, the entire initramfs userland (Rust, static - musl build) -- `boot/Dockerfile` — multi-stage: kernel → init → cpio → scratch image with - `/boot/vmlinuz-sandbox` + `/boot/initrd.img-sandbox` -- `os-image/` — VM images consuming the boot artifact: `base` (layered, - for builds), `rt` (base squashed to one layer — the default template in - examples), `python`, `python-rt`, `node`, `node-rt`, `browser`, - `android`, `e2b-rt` (rt plus e2b's envd, for the - e2b-compatible data plane), and `e2b-ci` (e2b-rt plus e2b's code interpreter) -- `scripts/` — `boot-bench.sh` (boot phase timing), `bench.sh` (the published - benchmark procedure), `sandboxd-e2e.sh` (bare-metal e2e, below), plus the - `archive`/`egress`/`intercept`/`socks`/`credentials`/`port` e2e drivers -- `packaging/` — the systemd unit deploy installs +```bash +make sandboxd +cocoon image pull ghcr.io/cocoonstack/sandbox/rt:24.04 + +cat >config.json <<'EOF' +{ + "listen": "127.0.0.1:7777", + "api_token": "change-me", + "pools": [ + {"template": "ghcr.io/cocoonstack/sandbox/rt:24.04", "net": "none", "size": "small", "warm": 2} + ] +} +EOF +dist/sandboxd -config config.json & + +# Wait for the golden snapshot and the warm pool. +until curl -sf -H 'Authorization: Bearer change-me' http://127.0.0.1:7777/v1/info | + jq -e '(.pools | length) > 0 and all(.pools[]; .golden and .warm >= .target)' >/dev/null; do sleep 1; done + +# Claim, exec and release three times; prints each latency. +(cd e2e && GOWORK=off go run ./cmd/demo -addr 127.0.0.1:7777 -token change-me \ + -template ghcr.io/cocoonstack/sandbox/rt:24.04) +``` -## Build & test +A stopped sandboxd leaves its VMs for the next start. To remove them: ```bash -make help # every target; the ones below are the daily set -make lint test # Rust: boot/init + silkd (fmt --check, clippy -D warnings, tests) -make go-lint # Go: protocol/wire + sandboxd + sdk/go + e2e + mcp, GOOS linux AND darwin -make go-test # Go: go test -race across the Go modules -make sh-lint # shellcheck every tracked shell script -make sandboxd # build dist/sandboxd -make bench # claim-tier + data-plane benchmarks on this node -make boot # kernel + initramfs artifact image (docker) - # KERNEL_MIRROR=… if kernel.org tarball paths 404 locally -make silkd-image # silkd release binary in a scratch carrier image -make images # base + python images against the local boot + silkd images +kill %1 +cocoon vm list --format json | jq -r '.[].config.name | select(startswith("sbx-"))' | + while read -r vm; do cocoon vm stop --force "$vm"; cocoon vm rm --force "$vm"; done ``` -The parent workspace's `go.work` includes these modules, which is why the -Makefile forces `GOWORK=off`: CI has no workspace, and a workspace build -resolves sibling checkouts instead of the pinned versions. silkd's integration tests spawn real -processes — run them in a Linux container too (`docker run rust:1 … cargo -test`) before touching platform-sensitive paths; macOS green alone has hidden -Linux-only breakage before. +`demo` is a small program over the [Go SDK](docs/sdk.md); the +[Python SDK](docs/sdk-python.md) has the same calls. [Deploying +sandboxd](docs/deploy.md) covers release binaries, the full config and systemd. -## Bare-metal e2e +## Related projects -`scripts/sandboxd-e2e.sh` drives the real stack on a node with cocoon and a -silkd-baked template image: golden build → warm pool → claim tiers → the -full v2-verb smoke (files/session/find/replace/watch/git/pty) → reap → -restart reconcile. +- [cocoon](https://github.com/cocoonstack/cocoon) — the microVM engine underneath +- [vk-sandbox](https://github.com/cocoonstack/vk-sandbox) — the Kubernetes provider that claims from sandboxd +- [sandbox-operator](https://github.com/cocoonstack/sandbox-operator) — the Kubernetes apiserver and e2b-compatible data plane over warm pools -```bash -TEMPLATE=rt:24.04 scripts/sandboxd-e2e.sh -# BRIDGE=br0 adds an egress pool and the egress lane-detect check; a plain -# `ip link add br0 type bridge` with no uplink is enough (NIC, not network). -# SANDBOXD_BIN/DEMO_BIN/SMOKE_BIN point at prebuilt binaries for nodes -# without a Go toolchain. -# VOLUME_IMAGE=/absolute/dataset.img enables the volumes proof (ro sharing; -# the image contains volume-e2e.txt); VOLUME_RW_IMAGE=/absolute/scratch.img -# adds the rw claim/release leg; VOLUME_LIVE=1 adds the live volume leg. -# Prebuilt runs also set VOLUME_SMOKE_BIN. -``` +## Layout + +- `silkd/` — in-guest product daemon (Rust), baked into the base image; + newline-JSON frames over vsock 2048 ([reference](docs/silkd.md)) +- `sandboxd/` — per-node control plane (Go): warm pools, the claim API and the + relays to silkd ([deploy](docs/deploy.md), [API](docs/sandboxd-api.md)) +- `sdk/go/` — [Go SDK](docs/sdk.md); `silkd/` is its connection layer and + `silkd/silkdtest/` a test fake +- `sdk/python/` — [Python SDK](docs/sdk-python.md), stdlib-only and synchronous +- `mcp/` — `sandbox-mcp`, an [MCP](docs/mcp.md) stdio server +- `sdk/openai/` — `cocoonstack-sandbox-openai`, a sandbox provider for the + [OpenAI Agents SDK](docs/openai-adapter.md) +- `sdk/langchain/` — `cocoonstack-sandbox-langchain`, a + [LangChain toolkit](docs/langchain.md) +- `protocol/wire/` — the frame vocabulary and `fixtures/`, the golden corpus the + Rust, Go and Python tests all round-trip, so wire drift fails CI +- `e2e/` — in-process full-stack tests (real pool/engine/relay/SDK, fake + cocoon + guest) and the [bare-metal drivers](docs/deploy.md#acceptance-drivers) +- `boot/kernel/` — kernel version pin (`VERSION` + matching tarball `SHA256`, + bump both together) and config fragments +- `boot/init/` — `sandbox-init`, the entire initramfs userland (static Rust) +- `boot/Dockerfile` — multi-stage: kernel → init → cpio → scratch image with + `/boot/vmlinuz-sandbox` + `/boot/initrd.img-sandbox` +- `os-image/` — VM images consuming the boot artifact: `base`, `rt`, `python`, + `python-rt`, `node`, `node-rt`, `browser`, `android`, `e2b-rt`, `e2b-ci` +- `scripts/` — benchmarks and bare-metal e2e, see + [Verifying a node](docs/deploy.md#verifying-a-node) +- `packaging/` — the systemd unit deploy installs ## CI @@ -162,32 +130,36 @@ cloud-hypervisor → exec /sbin/init (systemd, trimmed; cocoon-agent + silkd start at sysinit) ``` -Boot contract (cmdline keys consumed by sandbox-init): +The cmdline keys `sandbox-init` consumes are in the +[boot contract](docs/deploy.md#boot-contract). -| cmdline key | meaning | -|---|---| -| `cocoon.layers=a,b,…` | EROFS layer disks resolved from virtio-blk serials (or `/dev/vdX` paths, as the Firecracker lane passes them), lowerdir order | -| `cocoon.cow=x` | writable ext4 COW disk (same resolution rules) | -| `cocoon.timeout=10` | wait budget for the whole disk set, seconds (NIC MACs get a fixed 200 ms) | -| `cocoon.hostname=h` | set via `sethostname(2)` before handoff | -| `ip=addr::gw:mask:host:ethN:off[:dns0[:dns1]]` | cocoon CNI static config: persisted as a MAC-matched networkd unit in the new root (not applied in the initramfs); absent → the image's DHCP fallback covers the NIC | -| `sandbox.init=/path` | handoff target, default `/sbin/init` | -| `sandbox.debug=1` | fatal errors drop to `/bin/sh` (debug initramfs) instead of poweroff | -| `sandbox.trace=1` | emit one pre-handoff line with per-phase µs timings | +## Development -The pinned kernel builds no 8250 UART, so on cocoon's Firecracker lane -(`console=ttyS0`) the trace line, the debug shell and the fatal-error -message all go nowhere; they are visible on the Cloud Hypervisor lane's -virtio console. +```bash +make help # every target; the ones below are the daily set +make lint test # Rust: boot/init + silkd (fmt --check, clippy -D warnings, tests) +make go-lint # Go: protocol/wire + sandboxd + sdk/go + e2e + mcp, GOOS linux AND darwin +make go-test # Go: go test -race across the Go modules +make sh-lint # shellcheck every tracked shell script +make sandboxd # build dist/sandboxd +make bench # claim-tier + data-plane benchmarks on this node +make boot # kernel + initramfs artifact image (docker) + # KERNEL_MIRROR=… if kernel.org tarball paths 404 locally +make silkd-image # silkd release binary in a scratch carrier image +make images # base + python images against the local boot + silkd images +``` -`boot=cocoon-overlay` is ignored. Everything cocoon passes today keeps -working — images built here boot with an unmodified cocoon. +The parent workspace's `go.work` includes these modules, which is why the +Makefile forces `GOWORK=off`: CI has no workspace, and a workspace build +resolves sibling checkouts instead of the pinned versions. silkd's integration +tests spawn real processes — run them in a Linux container too (`docker run +rust:1 … cargo test`) before touching platform-sensitive paths; macOS green +alone has hidden Linux-only breakage before. ## License The server stack — sandboxd, silkd, the boot chain, the OS images, and the -MCP server — is licensed under [AGPL-3.0](LICENSE). The client SDKs +MCP server — is a product, and is licensed under [AGPL-3.0](LICENSE). The client SDKs (`sdk/go`, `sdk/python`, `sdk/openai`, `sdk/langchain`) and the `protocol/wire` module the Go SDK requires are licensed under Apache-2.0 -(see the `LICENSE` file in each directory), so embedding a client in a -proprietary agent stack carries no copyleft obligation. +(see the `LICENSE` file in each directory) and carry no copyleft obligation. diff --git a/docs/deploy.md b/docs/deploy.md index 68c1697c3..11ad42158 100644 --- a/docs/deploy.md +++ b/docs/deploy.md @@ -15,11 +15,10 @@ the cocoon CLI and needs a template image with silkd baked in. reads. sandboxd logs a warning at startup when the detected cocoon is below v0.6.9 (a dev/`master-` build is assumed current); changing volumes on a running claim needs v0.7.2, and a pool's `reader` needs v0.7.3 -- The sandbox boot artifact installed where cocoon finds it - (`/boot/vmlinuz-sandbox`, `/boot/initrd.img-sandbox` — from - `ghcr.io/cocoonstack/sandbox/boot:`) - A silkd-baked template image, e.g. `ghcr.io/cocoonstack/sandbox/base:24.04` - (pull via cocoon, or `cocoon image import` a tar) + (pull via cocoon, or `cocoon image import` a tar). The image carries the boot + artifact (`/boot/vmlinuz-sandbox`, `/boot/initrd.img-sandbox`), so the host + needs neither file Prebuilt static linux/amd64 and linux/arm64 binaries (`sandboxd`, `sandboxd.dbg`, `sandbox-mcp`, `silkd`, with `checksums.txt`) ship with every @@ -29,6 +28,10 @@ multi-arch manifests (`browser`, `android`, `e2b-rt` and `e2b-ci` remain amd64-only). Build from source with `make sandboxd` (produces `dist/sandboxd`); either way `sandboxd -version` reports what you are running. +`base` is layered, for builds. `rt` is `base` squashed to one layer and is the +default template in the examples; `python-rt` and `node-rt` are the squashed +flavors. + ## Upgrading This release does not convert VM or snapshot state from older releases. Drain old @@ -967,21 +970,57 @@ curl -s -H "Authorization: Bearer $TOKEN" http://127.0.0.1:7777/v1/info | jq . ``` The repository's `scripts/sandboxd-e2e.sh` runs the full loop on a real node -(golden build → warm pool → claim tiers → the complete verb smoke → reap → -restart reconcile); set `BRIDGE=` to include the egress lane. +with cocoon and a silkd-baked template image (golden build → warm pool → claim +tiers → the complete verb smoke, files/session/find/replace/watch/git/pty → +reap → restart reconcile): + +```bash +TEMPLATE=rt:24.04 scripts/sandboxd-e2e.sh +``` + +Set `BRIDGE=` to include the egress lane: it adds an egress pool and the +lane-detect check. A plain `ip link add br0 type bridge` with no uplink is +enough, since the check needs a NIC, not a network. On a node without a Go +toolchain, point `SANDBOXD_BIN`, `DEMO_BIN`, and `SMOKE_BIN` at prebuilt +binaries. To include the read-only volume proof, put a nonempty `volume-e2e.txt` in the filesystem image and run `VOLUME_IMAGE=/srv/datasets/imagenet.img -scripts/sandboxd-e2e.sh`. The script verifies two concurrent mounts, read-only -enforcement, warm-pool consumption, and an unchanged source checksum. On a node -using prebuilt binaries, also set `VOLUME_SMOKE_BIN` beside `SANDBOXD_BIN`, -`DEMO_BIN`, and `SMOKE_BIN`. +scripts/sandboxd-e2e.sh` (the path must be absolute). The script verifies two +concurrent mounts, read-only enforcement, warm-pool consumption, and an +unchanged source checksum. On a node using prebuilt binaries, also set +`VOLUME_SMOKE_BIN` beside `SANDBOXD_BIN`, `DEMO_BIN`, and `SMOKE_BIN`. Add `VOLUME_RW_IMAGE=/srv/datasets/scratch.img` (a second, writable image) to also run the writable leg: a durable write across release, second-writer exclusion, and a clean read-only claim afterward. `VOLUME_LIVE=1` adds the live volume leg on cocoon v0.7.2 or newer. +### Acceptance drivers + +The bare-metal drivers live under `e2e/cmd/`, and a script in `scripts/` runs +some of them on a node: + +| driver | proves | script | +|---|---|---| +| `demo`, `smoke` | claim tiers and the verb surface | `sandboxd-e2e.sh` | +| `volumesmoke` | shared read-only, writable, and live volumes | `sandboxd-e2e.sh` | +| `meshsmoke` | template routing on a two-node mesh | | +| `crossnode` | a checkpoint claimed on another node through the shared store | | +| `coldproof` | a cold claim, by the guest's own clock | | +| `lifecycle` | idle → hibernate → archive → wake | `archive-e2e.sh` | +| `egresssmoke` | guarded egress on either lane | `egress-e2e.sh` | +| `interceptsmoke` | HTTPS interception | `intercept-e2e.sh` | +| `sockssmoke` | the SOCKS5 door | `socks-e2e.sh` | +| `credsmoke` | claim credential injection | `credentials-e2e.sh` | +| `portsmoke`, `guestserver` | the guest-port relay; `guestserver` is its guest listener | `port-e2e.sh` | +| `ringsmoke` | the output ring cap and exec as a user | | +| `androidsmoke`, `browsersmoke` | the android and browser flavors | | +| `pullbench`, `pushbench`, `rpcbench`, `qaab` | performance measurements | | + +`scripts/bench.sh` is the published [benchmark](benchmarks.md#reproduce) +procedure and `scripts/boot-bench.sh` times the boot phases. + ## TLS for SDK clients sandboxd serves plain HTTP behind a TLS-terminating proxy. Configure a stable @@ -1091,3 +1130,31 @@ sandboxd: a preview URL. - **Health**: `GET /healthz` on `preview_listen` answers 200 `ok`, for the load balancer's check. + +## Boot contract + +`sandbox-init` is `/init` in the boot artifact's initramfs. cocoon builds the +kernel cmdline, and `sandbox-init` consumes these keys: + +| cmdline key | meaning | +|---|---| +| `cocoon.layers=a,b,…` | EROFS layer disks resolved from virtio-blk serials (or `/dev/vdX` paths, as the Firecracker lane passes them), lowerdir order | +| `cocoon.cow=x` | writable ext4 COW disk (same resolution rules) | +| `cocoon.timeout=10` | wait budget for the whole disk set, seconds (NIC MACs get a fixed 200 ms) | +| `cocoon.hostname=h` | set via `sethostname(2)` before handoff | +| `ip=addr::gw:mask:host:ethN:off[:dns0[:dns1]]` | cocoon CNI static config: persisted as a MAC-matched networkd unit in the new root (not applied in the initramfs); absent → the image's DHCP fallback covers the NIC | +| `sandbox.init=/path` | handoff target, default `/sbin/init` | +| `sandbox.debug=1` | fatal errors drop to `/bin/sh` (debug initramfs) instead of poweroff | +| `sandbox.trace=1` | emit one pre-handoff line with per-phase µs timings | + +The pinned kernel builds no 8250 UART, so on cocoon's Firecracker lane +(`console=ttyS0`) the trace line, the debug shell and the fatal-error +message all go nowhere; they are visible on the Cloud Hypervisor lane's +virtio console. + +`boot=cocoon-overlay` is ignored. Everything cocoon passes today keeps +working — images built here boot with an unmodified cocoon. + +The kernel's config in `boot/kernel/` is a fragment. On amd64 it merges over +`x86_64_defconfig` + `kvm_guest.config`; on arm64 it merges over `defconfig`, +then `sandbox-arm64.config`. diff --git a/docs/index.md b/docs/index.md index d3aaec111..af0ccd2cb 100644 --- a/docs/index.md +++ b/docs/index.md @@ -19,7 +19,8 @@ vsock-only I/O (hardened default); `net=egress` attaches a bridge/CNI NIC. ## Guides - [Deploying sandboxd](deploy.md) — single node, configuration reference, - images, running as a service + images, running as a service, the bare-metal e2e and acceptance drivers, the + boot contract - [Clusters](cluster.md) — joining a mesh, querying members, how redirect placement works, relocating handles, and surviving node loss (archived and replicated claims)