bug(docker): recovery stop aborts when supervisor-failure watch restores Error #4727

User Story

I use OpenShell's Docker driver for isolated development sandboxes on ARM64 Linux. While testing recovery from supervisor loss in my deployment, sandbox stop intermittently failed even though the driver finished stopping the workload. I need the supported stop/start recovery path to preserve the sandbox and its workspace without deleting it or editing the database.

Problem Statement

During recovery from ControlSupervisorExited, an explicit stop moves the sandbox from Error to Stopping. A driver-watch snapshot can then change it back to Error before the stop operation completes. The workload is stopped, but the CLI returns:

Error: × code: 'The operation was aborted', message:
"sandbox lifecycle changed while stop completed"

The persisted phase remains Error. This reproduced with unchanged upstream development binaries in one of two fresh disposable probes, as well as in the original recovery test. The other fresh probe completed its first stop successfully, so this is timing-dependent rather than a deterministic reproducer.

Related to #4115. This is a focused Docker recovery-stop case within that lifecycle-coordination audit.

Impact / Why This Matters

Recovery automation receives a failed stop and cannot proceed through the normal stop/start sequence. In the observed failures, refetching the same sandbox after workload cleanup and explicitly retrying stop succeeded; start then returned the same sandbox ID to Ready, with a marker file intact. That workaround required an extra recovery decision after an apparently failed operation. No sandbox deletion or database modification was needed.

Acceptance Criteria

• A stop accepted from an ordinary infrastructure-error state completes as Stopped after the driver confirms that the same workload has stopped, despite an overlapping supervisor-failure observation.
• A subsequent start preserves sandbox identity and workspace contents and reaches Ready.
• Genuine stop failures, supervisor-loss containment, and protection against concurrent start/delete or newer runtime generations remain intact.
• Regression coverage forces the failure-snapshot/stop-completion overlap; timing-based stress alone is not sufficient.

Reproduction Steps

Use a working Docker-backed OpenShell deployment and a disposable sandbox. The probes used an existing custom Nix development image and its working non-root policy, with no caller-supplied volume mounts. A stock workload image has not been tested for this report.

1. Create a long-running sandbox using that deployment's workload image and policy, then wait for Ready:

   name=stop-race-probe
   openshell sandbox create --name "$name" \
     --policy ./working-policy.yaml --no-auto-providers --detach -- \
     /bin/bash -lc 'exec sleep infinity'
   openshell sandbox get "$name"

2. On the Docker host, select only this sandbox's supervisor. Set namespace to the Docker driver's configured sandbox_label:

   namespace=openshell-v014-trial
   supervisor=$(docker ps \
     --filter "label=openshell.ai/managed-by=openshell" \
     --filter "label=openshell.ai/sandbox-namespace=$namespace" \
     --filter "label=openshell.ai/sandbox-name=$name" \
     --filter "label=openshell.ai/isolation-role=supervisor" \
     --format '{{.ID}}')

3. Verify that the result is exactly one container belonging to the disposable sandbox, then terminate only that supervisor:

   docker inspect "$supervisor"
   docker kill --signal KILL "$supervisor"

4. Poll openshell sandbox get "$name" --output json until it reports Error with ControlSupervisorExited. Immediately run openshell sandbox stop "$name" while the driver's workload cleanup is still in progress.

5. Intermittently, stop returns the Aborted error above, and another get still reports Error. Gateway logs show Stopping -> Error before the driver reports that workload cleanup finished.

6. After cleanup, explicitly retry openshell sandbox stop "$name", then openshell sandbox start "$name". In the reproduced case this retained the ID and marker file and reached Ready.

Both disposable probes were deleted after collecting evidence. Their containers and volumes were removed; the existing development sandbox stayed Ready with its marker hash unchanged.

Environment

• CLI and gateway: 0.1.5-dev.71+g1cb9ef8ab, commit 1cb9ef8ab469354a641893f8e26957e988f5c7ef.
• Upstream prebuilt ARM64 CLI and matching gateway/supervisor/sandbox images; no source patches or scheduling instrumentation.
• Single gateway, Docker driver, SQLite, host networking, TLS and user mTLS enabled; not an HA deployment.
• NixOS 26.11, aarch64, in OrbStack; kernel 7.2.9-orbstack-00391-g2b298d05fbd4.
• Docker Engine 29.8.1.
• Latest stable checked: v0.1.4. At investigation time, both main and the dev release resolved to the tested commit.

Pinned ARM64 images:

ghcr.io/nvidia/openshell/gateway@sha256:8c0d570c43f49aceed6c0ede8af89f93ec34b60e365b172c85aae80911ed9093
ghcr.io/nvidia/openshell/supervisor@sha256:8ebf43166550876297ab85bd48ed1a8fb8873d7b906427615beca5b91b66f8b8
ghcr.io/nvidia/openshell/sandbox@sha256:a911a794bb7d898110f0d1778ad99679e86df0b0bcca283f06e1902d1418ff9d

Logs

Relevant events from the failing fresh probe, UTC on 2026-10-19. Sandbox ID: 49ad9479-3b47-4f24-8693-8adf7bfd1808. Fields unrelated to the failure are omitted.

21:26:34.134456  test: supervisor killed with SIGKILL
21:26:35.138546  Docker supervisor container exited unexpectedly; reason=ControlSupervisorExited; exit=137
21:26:35.373484  test: sandbox get reports Error
21:26:37.027779  driver_watch: Sandbox phase changed old_phase=Stopping new_phase=Error
21:26:45.250140  Stopped Docker sandbox after control supervisor failure
21:26:45.273746  test: first stop exits 1; Aborted: sandbox lifecycle changed while stop completed
21:26:45.360268  test: sandbox get still reports Error
21:26:46.493840  test: explicit stop retry succeeds
21:26:46.574456  test: sandbox get reports Stopped
21:26:47.376087  test: start completed; same sandbox reports Ready
21:26:47.486618  test: retained marker file verified

Agent Investigation

The captured phase transition and the pinned source point to a gap between failure observation and stop completion:

• handle_docker_runtime_failure records/publishes the supervisor failure before awaiting workload stop. While the workload is still running, the driver's snapshot path can continue reporting that failure.
• apply_driver_snapshot preserves Stopping for several observations, but permits an incoming error to replace it.
• commit_sandbox_stopped accepts Stopping or an already Stopped record and aborts on Error, even after the driver stop returned successfully.

#4231 and #4395 are both merged and included in the tested commit. #4231 permits stop from Error; #4395 retries resource-version conflicts while the record remains Stopping. Neither covers the observed phase regression. #4235, also included, handles late exit observations across restart, a different transition.

I searched open and closed issues for the exact error and related stop/supervisor failures and did not find an exact matching report. #4115 is the broader tracking issue.