perf(driver-docker): 250 ms supervisor healthcheck costs ~12% of a CPU core per idle sandbox #4729

Summary

With the Docker compute driver, every sandbox's supervisor container gets a Docker healthcheck that runs every 250 ms, both during the start period and after it. Each check is a docker exec of /openshell-supervisor health --socket /run/openshell/health.sock, so an idle sandbox triggers four execs per second for its whole lifetime. On our hosts this costs about 12.5% of one CPU core per idle sandbox in the supervisor's cgroup. dockerd and containerd add more on top, about 5% each with two idle sandboxes.

The driver reads the health status only while waiting for startup readiness (poll_docker_supervisor_ready). After the supervisor is healthy, nothing in the driver depends on a fast interval. That makes the steady-state 250 ms cost avoidable.

Environment

• OpenShell v0.1.4 (deb install, user-level gateway), Docker compute driver
• Docker Engine 29.9.1, Ubuntu 24.04, kernel 6.8.0, x86_64 (4 vCPU / 8 GB VM)
• The constant is unchanged on main as of 2026-10-19

Reproduction

1. Install OpenShell with the Docker driver and create two sandboxes:

   openshell sandbox create --name a --detach
   openshell sandbox create --name b --detach
   

2. Leave them idle: no client connected, nothing running.
3. Inspect the supervisor container's healthcheck:

   docker inspect -f '{{json .Config.Healthcheck}}' <supervisor container>
   

   Result:

   {"Test":["CMD","/openshell-supervisor","health","--socket","/run/openshell/health.sock"],
    "Interval":250000000,"Timeout":2000000000,"StartPeriod":60000000000,"StartInterval":250000000,"Retries":3}

4. Measure the CPU each supervisor cgroup uses over 30 s (usage_usec delta from /sys/fs/cgroup/system.slice/docker-<id>.scope/cpu.stat).

Measurements (idle sandboxes)

docker stats showed the same thing across five samples: 11–17% per idle supervisor container. The long-running supervisor process is nearly idle; the cost is the short-lived health execs started in the container's cgroup four times a second.

At density this adds up. Twenty idle sandboxes would spend roughly 2.5 cores on healthchecks in the supervisor cgroups alone, before the daemons' share.

Where it comes from

crates/openshell-driver-docker/src/lib.rs (v0.1.4):

const SUPERVISOR_HEALTH_INTERVAL_NS: i64 = 250_000_000;
const SUPERVISOR_HEALTH_TIMEOUT_NS: i64 = 2_000_000_000;
const SUPERVISOR_HEALTH_START_PERIOD_NS: i64 = 60_000_000_000;
...
healthcheck: Some(HealthConfig {
    ...
    interval: Some(SUPERVISOR_HEALTH_INTERVAL_NS),
    start_period: Some(SUPERVISOR_HEALTH_START_PERIOD_NS),
    start_interval: Some(SUPERVISOR_HEALTH_INTERVAL_NS),
}),

The only reader of the health status we found is poll_docker_supervisor_ready, which waits for HEALTHY at startup.

Suggested fix

• Keep fast startup, slow the steady state: leave start_interval at 250 ms so readiness is detected quickly, and set interval to something like 30 s. Docker uses start_interval only during the start period, so startup latency is unchanged.
• Make it configurable: expose the steady-state interval in the driver config, for example [openshell.drivers.docker] supervisor_health_interval, for operators who want a different trade-off.
• Or drop the periodic check after readiness: if the gateway already detects a lost supervisor through its session, the steady-state healthcheck may not be needed at all.

Related

• perf(sandbox): idle openshell-sandbox wakes ~890 times/s (orphan-reaper thread) #3756 (closed) covered idle wake-ups in the in-sandbox openshell-sandbox process, about 2% per sandbox. This report is a separate, larger cost in the supervisor container under the Docker driver.