Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
18 changes: 18 additions & 0 deletions Cargo.lock

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

2 changes: 1 addition & 1 deletion architecture/compute-runtimes.md
Original file line number Diff line number Diff line change
Expand Up @@ -211,7 +211,7 @@ delete, reconciliation removes the row; otherwise it can remain `Deleting`.

| Runtime | Best fit | Sandbox boundary | Notes |
|---|---|---|---|
| Docker | Local development with Docker available. | Container plus nested sandbox namespace. | Uses host networking so loopback gateway endpoints work from the supervisor. Advertises the combined-supervisor policy-DNS and transparent-TCP substrate. |
| Docker | Local Linux development with Docker available. | Network-disabled workload container plus native host supervisor. | Uses RFC 0012 supervisor-owned creation and OCI seccomp notification to carry policy DNS and transparent TCP without a proxy port or an OpenShell binary in the workload. |
| Podman | Rootless or single-machine deployments. | Container plus nested sandbox namespace. | Uses the Podman REST API and CDI GPU devices when available. Delivers the supervisor via OCI image volume by default; falls back to extracting the binary to a host-side cache and bind-mounting it when `userns` is configured (overlay does not support idmapped mounts). Advertises the combined-supervisor policy-DNS and transparent-TCP substrate. |
| Kubernetes | Cluster deployment through Helm. | Pod plus nested sandbox namespace. | Uses Kubernetes API objects, service accounts, secrets, PVC-backed workspace storage, and GPU resources. |
| VM | Experimental microVM isolation. | Per-sandbox libkrun VM. | Managed endpoint-backed driver. The gateway spawns `openshell-driver-vm`, waits for its Unix socket, and then consumes it through the same remote `compute_driver.proto` path used by unmanaged endpoint drivers. The VM driver boots a cached bootstrap `rootfs.ext4`, prepares requested OCI images inside a bootstrap VM with `umoci`, attaches the prepared image disk read-only, and gives each sandbox a writable `overlay.ext4` for merged-root changes and runtime material. The driver persists each accepted launch request beside the overlay and restarts those VMs on driver startup without recreating the overlay. |
Expand Down
8 changes: 8 additions & 0 deletions architecture/sandbox.md
Original file line number Diff line number Diff line change
Expand Up @@ -139,6 +139,14 @@ generation-pinned authorization form the transparent TCP security boundary.
Docker and Podman do not currently advertise usable IPv6 egress for this
substrate, so AAAA queries return NOERROR/NODATA and IPv6 DNS remains fenced.

Isolation backends may provide the same substrate without namespace listeners.
Their DNS source carries bounded wire exchanges directly to policy DNS, and
their connection source supplies the captured synthetic destination and calling
binary identity with each stream. The Docker host-supervisor backend uses OCI
seccomp user notification to inject those streams while Docker networking stays
disabled. The supervisor therefore binds no workload-visible proxy or DNS port;
the synthetic mapping and pinned-destination checks remain unchanged.

Provider credential placeholders are resolved through the live provider state
for each HTTP request, after destination and L7 policy admission. A static
credential resolves only when the request host, port, and path match an endpoint
Expand Down
9 changes: 9 additions & 0 deletions crates/openshell-driver-docker/Cargo.toml
Original file line number Diff line number Diff line change
Expand Up @@ -16,6 +16,7 @@ path = "src/main.rs"

[dependencies]
openshell-core = { path = "../openshell-core", default-features = false, features = ["driver-extraction"] }
openshell-isolation = { path = "../openshell-isolation" }
openshell-otel = { path = "../openshell-otel" }

opentelemetry = { workspace = true }
Expand All @@ -29,12 +30,20 @@ tracing-subscriber = { workspace = true }
bytes = { workspace = true }
serde = { workspace = true }
serde_json = { workspace = true }
sha2 = { workspace = true }
prost-types = { workspace = true }
bollard = { version = "0.20" }
url = { workspace = true }
clap = { workspace = true }
miette = { workspace = true }
toml = { workspace = true }
async-trait = "0.1"
base64 = { workspace = true }
uuid = { workspace = true }

[target.'cfg(target_os = "linux")'.dependencies]
libc = "0.2"
nix = { workspace = true, features = ["socket", "uio"] }

[dev-dependencies]
openshell-otel-test-support = { path = "../openshell-otel-test-support" }
Expand Down
241 changes: 63 additions & 178 deletions crates/openshell-driver-docker/README.md
Original file line number Diff line number Diff line change
@@ -1,193 +1,78 @@
# openshell-driver-docker

Docker-backed compute driver for local OpenShell gateways.

When the gateway configures `[openshell.gateway.otlp]`, Docker compute-driver
spans export to the same OTLP/gRPC collector with the service name
`openshell-driver-docker`. The in-process driver preserves the gateway trace
context and emits the compute-driver RPC boundary that a standalone driver
would expose.

The driver manages sandbox containers through the local Docker daemon with the
`bollard` client. It is intended for developer environments where Docker is
already available and running Kubernetes would be unnecessary.
Docker-backed compute and isolation driver for local Linux OpenShell gateways.

The driver connects to `[openshell.drivers.docker].socket_path` when configured.
Otherwise, it uses the first standard local Docker socket that responds to an
API ping, which is the same selection mechanism used by gateway auto-detection.
An explicitly selected Docker driver falls back to `/var/run/docker.sock` when
no candidate responds.

## Runtime Model

The gateway runs as a host process. The Docker driver creates one container per
sandbox and starts the `openshell-sandbox` supervisor inside that container. The
supervisor then creates the nested sandbox namespace for the agent process.

## Stop and Start

Stop stops the managed container without removing it. Docker retains the
container writable layer, attached volumes, labels, token material, and restart
policy. Start starts that same container, so files in the resolved OCI
workspace remain available. A durably stopped sandbox is excluded from
gateway startup recovery and stays stopped across gateway restarts. Delete
continues to force-remove the container and clean up driver-owned material.
Graceful gateway shutdown sends `StopSandbox` for each sandbox whose persisted
phase requires running compute without changing that persisted intent. On
startup, the gateway sends an idempotent `StartSandbox` request for the same
sandboxes, restarting their retained containers. Explicitly stopped sandboxes
remain excluded.

Before creating the container, the driver inspects the final sandbox image and
captures its immutable image ID, raw OCI `Config.User`, and OCI
`Config.WorkingDir`. Container creation uses that image ID, preventing a
mutable tag from changing between inspection and launch. The supervisor runs as
root, resolves omitted policy identity fields from the image declaration, and
drops only agent children to the resulting identity. Named OCI components
remain names after validation; a missing group is filled with the user's
numeric primary GID. Explicit `process.run_as_user` and
`process.run_as_group` values take precedence independently.

An absolute OCI working directory becomes the agent workspace. An empty,
root (`/`), or explicit `/sandbox` declaration uses `/sandbox`, which OpenShell
creates when necessary and owns as a compatibility workspace. Any other image
workdir must already exist without symlink components. The completed identity,
including supplementary groups, must already be able to traverse every parent
and write and enter the workdir. OpenShell does not change its ownership or
mode.

OpenShell deliberately asks the Linux kernel to make this access decision
under the completed sandbox identity instead of reproducing permission rules
from ownership and mode bits. Mode-bit inspection alone can reject authority
granted by a POSIX ACL or overlook a denial imposed by a Linux Security Module
such as SELinux or AppArmor. OpenShell does not configure or otherwise manage
ACLs or LSM policy here; the one-shot validator only observes the kernel's
effective decision. This keeps the no-authority-expansion invariant aligned
with the access the eventual workload will receive without adding a separate,
incomplete permission model to OpenShell.

Image `VOLUME` declarations must not cover the workdir or one of its parents
because Docker would mount the volume before the supervisor could validate the
immutable image path.
Workdirs under the standard OCI runtime namespaces `/proc`, `/sys`, and `/dev`
are rejected, as are paths that overlap concrete OpenShell control resources.
The workspace is the child cwd and `HOME`. The supervisor starts from `/`, then
reports an invalid workdir as a readiness failure.

Docker containers join an OpenShell-managed bridge network. The driver injects
`host.openshell.internal` and `host.docker.internal` so supervisors have stable
names for reaching the gateway host. On Docker Desktop, Colima, Rancher
Desktop, OrbStack, and macOS-hosted gateways, those names use Docker's
`host-gateway` alias. The driver requests a separate IPv4 loopback callback
listener when the primary listener does not already cover it. On native Linux
Docker, the gateway also binds the bridge gateway IP so containers can call
back to the host process.

## Container Contract

The driver-controlled container settings are part of the sandbox security
contract:

| Setting | Purpose |
|---|---|
| `user = "0"` | The supervisor needs root inside the container to prepare namespaces, mounts, Landlock, and seccomp. |
| `network_mode = openshell` | Places the supervisor on the managed Docker bridge network. |
| `cap_add` | Grants supervisor-only capabilities required for namespace setup and process inspection. |
| `apparmor=unconfined` | Avoids Docker's default profile blocking required mount operations. |
| `restart_policy = no` | A canonical main-process exit remains terminal and is not silently restarted by Docker. |
| `PidsLimit` | Enforces the sandbox PID budget at the Docker cgroup layer. Set `[openshell.drivers.docker].sandbox_pids_limit = 0` to inherit the Docker/runtime default. |
| CDI GPU request | Uses opaque `driver_config.cdi_devices` values when set; otherwise selects the requested count of NVIDIA CDI GPUs in round-robin order when daemon CDI support is detected. Docker daemon `/info` can permit `nvidia.com/gpu=all` as a WSL2 all-only compatibility fallback, where it counts as one selectable device. Exact CDI device lists must not contain duplicates and must match the effective GPU count. |
| `policy-dns-transparent-tcp` capability | Declares that the combined Docker supervisor can own namespace-local DNS/TCP capture and coupled workload restart. The shared supervisor still owns DNS eligibility, mappings, authorization, pinned dialing, relaying, and OCSF decisions. The marker is stripped from the workload environment. |

The agent child process does not retain these supervisor privileges.

## Driver Config Mounts

The gateway forwards the `docker` block from `--driver-config-json` to this
driver. The driver accepts user-supplied `mounts` entries with these Docker
mount types:

- `bind`: mounts an absolute host path when `[openshell.drivers.docker]`
has `enable_bind_mounts = true`.
- `volume`: mounts an existing Docker named volume. The driver validates that
the volume exists before provisioning and never creates or removes it.
Docker local-driver volumes created with bind options are treated as host
bind mounts and require `enable_bind_mounts = true`.
- `tmpfs`: mounts an in-memory filesystem with optional `options`,
`size_bytes`, and `mode`.

Host bind mounts are disabled by default because they expose gateway host
paths to sandbox requests. Image mounts are not part of the Docker
driver-config schema. The driver still uses internal bind mounts for
OpenShell-owned supervisor, token, and TLS material.

Docker `bind` mounts accept `source`, `target`, optional `read_only`, and an
optional `selinux_label` of `shared` (applies `:z`) or `private` (applies
`:Z`) for SELinux-enforcing hosts. Docker `volume` mounts may include
`subpath`. User-supplied bind and volume mounts are read-only by default; set
`read_only: false` to make them writable. Mount `source`, `target`, and
`subpath` values must not contain surrounding whitespace. Mount targets must be
absolute container paths and must not replace or contain the resolved workspace
root. Nested workspace mounts remain valid. Mounts also must not overlap the
configured SSH socket or the reserved `/opt/openshell`, `/etc/openshell`,
`/etc/openshell-tls`, `/run/openshell`, `/run/openshell-sidecar`, and network
namespace roots.

Example named-volume usage:

```shell
docker volume create openshell-work

openshell sandbox create \
--driver-config-json '{"docker":{"mounts":[{"type":"volume","source":"openshell-work","target":"/sandbox/work"}]}}' \
-- claude
```

## Supervisor Binary Resolution

The Docker driver bind-mounts a host-side Linux `openshell-sandbox` binary into
each sandbox container. Resolution order is:
Otherwise, it selects a standard local Docker socket and falls back to
`/var/run/docker.sock` when Docker is explicitly enabled.

## Runtime model

Docker uses RFC 0012 supervisor-owned boundary creation. The compute driver
resolves the immutable workload image and launches a native
`openshell-sandbox` process on the gateway host. That supervisor registers the
Docker `IsolationBackend`, creates the container, confirms enforcement, starts
the workload, and exposes exec through the normal supervisor relay.

The workload container contains no OpenShell binary, gateway credential, TLS
private key, or supervisor capability set. The dedicated mode runs the
workload as the gateway user's numeric UID and GID so the unprivileged host
supervisor can capture syscall arguments and hash the calling executable. It
uses `/` as the initial working directory and does not provide driver mounts.

The backend binds a private host Unix socket before container creation and
passes OCI seccomp `listenerPath`, `listenerMetadata`, and `SCMP_ACT_NOTIFY`
through the Docker seccomp profile. runc sends the listener FD directly to the
host supervisor with `SCM_RIGHTS` before the workload starts. The container
drops every Linux capability and has direct networking disabled.

The listener injects connected sockets for workload TCP and DNS operations.
DNS queries travel directly to the supervisor policy-DNS service and return
short-lived synthetic addresses. A later connection to a synthetic address is
bound to the queried hostname, calling-binary identity, allowed port, policy
generation, and pinned real addresses. The supervisor consumes these streams
without binding a container-visible proxy port or setting proxy environment
variables. Direct real-IP and non-mediated UDP traffic fail closed.

For TLS termination, the backend read-only mounts the generated public CA and
combined public trust bundle. CA private key material remains in host
supervisor memory.

The current implementation supports create, confirm, start, wait,
signal-main-process, delete, Docker exec, policy DNS, transparent TCP, and TLS
termination. Port forwarding, exec signaling, GPU devices, resource limits,
driver mounts, images that require root, and durable running-boundary recovery
remain unsupported and fail closed.

## Supervisor binary resolution

The native host supervisor is resolved in this order:

1. `supervisor_bin` in `[openshell.drivers.docker]`.
2. `supervisor_image` in `[openshell.drivers.docker]`, extracting
`/openshell-sandbox` from that image.
3. A sibling `openshell-sandbox` next to the running `openshell-gateway` binary.
`/openshell-sandbox` to a host cache.
3. A sibling `openshell-sandbox` next to `openshell-gateway`.
4. A local Linux cargo target build for the Docker daemon architecture.
5. The release-matched default supervisor image, extracting `/openshell-sandbox`.

Release and Docker-image gateway builds bake the matching supervisor image tag
into the binary at compile time. The default Docker supervisor image is not
`:latest` unless a custom build explicitly sets that tag.

## Callback and TLS
5. The release-matched default supervisor image.

`OPENSHELL_ENDPOINT` is injected from the gateway's configured gRPC endpoint.
When no endpoint is configured, the driver uses
`host.openshell.internal:<gateway-port>` with the appropriate HTTP or HTTPS
scheme. Set `host_gateway_ip` only when the host has an explicit, locally
assigned address that containers should use for callbacks; package-managed
macOS gateways should leave it unset.
The resolved binary executes on the host; it is never mounted into the
workload container.

For HTTPS endpoints, the server certificate must include the endpoint host as a
subject alternative name. Docker sandboxes also need the client TLS bundle
mounted into the container and exposed with:
## Gateway authentication

- `OPENSHELL_TLS_CA`
- `OPENSHELL_TLS_CERT`
- `OPENSHELL_TLS_KEY`
The compute driver writes the sandbox JWT to a host-only state directory and
passes that path only to the native supervisor. HTTPS CA, certificate, and key
paths likewise remain on the host. The supervisor connects to the gateway over
the configured `grpc_endpoint`; host aliases such as
`host.openshell.internal` are normalized to loopback for the native process.

HTTP endpoints reject TLS material because the supervisor would not use it.
## Testing

## Environment Ownership
The standard Docker runner exercises this implementation:

The driver merges template environment and sandbox spec environment first, then
overwrites security-critical keys:

- `OPENSHELL_ENDPOINT`
- `OPENSHELL_SANDBOX_ID`
- `OPENSHELL_SANDBOX`
- `OPENSHELL_SSH_SOCKET_PATH`
- `OPENSHELL_MAIN_PROCESS_SPEC`
- TLS path variables when HTTPS is enabled
```shell
mise run e2e:docker
```

Do not allow sandbox images or templates to override these values.
Set `OPENSHELL_E2E_SANDBOX_IMAGE` to test another workload image supported by
the smoke scenario.
Loading
Loading