bug(gateway): a sandbox whose isolation backend selection changed stays in Starting and never reports the mismatch #4737

User Story

I maintain an out-of-tree OpenShell compute driver that runs workloads behind a registered isolation backend. When I change or remove a backend registration, the affected sandboxes need to show why they can't start. Today they sit in Starting with a status that says a restart is scheduled, and the real cause appears only in gateway logs.

Problem Statement

Each sandbox is pinned to the isolation backend that was selected when it was created (internal.openshell.ai/isolation-backend). If that pin no longer matches the gateway's selection, only an interactive sandbox start reports the mismatch to the user. The automatic paths in crates/openshell-server/src/compute/mod.rs log it and skip the sandbox:

• Automatic restart: restart_sandbox_runtime returns on the error from validate_isolation_backend_binding before it claims the restart or writes status. restart_due_sandboxes then logs "Automatic sandbox restart attempt failed" on every scan, every RESTART_SCAN_INTERVAL (500 ms). The Ready condition keeps the reason written when the main process exited (MainProcessExitDraining or MainProcessRestartScheduled), which says a restart is coming. No deadline ends this state: apply_main_process_restart does not arm a new provisioning deadline, and the sandbox's existing one already finished at Ready, so the provisioning-deadline sweep never expires it.
• Startup recovery: start_persisted_sandboxes_with_authentication logs "Rejected recovered isolation backend selection" and continues, leaving the sandbox's phase and reason unchanged.
• Lifecycle recovery: recover_persisted_lifecycle_transitions logs "Rejected pending isolation backend selection" for a sandbox in Starting and continues. That sandbox has an armed preparation deadline, so it later moves to Error with ImagePreparationTimedOut, a reason that points at image preparation instead of the backend.

Impact / Why This Matters

• Stuck sandboxes with a wrong reason: after an operator renames or removes a backend registration and restarts the gateway, every sandbox with an automatic restart in progress stays in Starting indefinitely. openshell sandbox get reports that a restart is scheduled.
• Misdiagnosis: a sandbox caught mid-start by lifecycle recovery fails with an image-preparation timeout, so the user investigates the image instead of the gateway configuration.
• Log noise: the gateway logs a warning for each stuck sandbox twice a second.
• Hidden from users: users find the real cause only in gateway logs, which they often can't read.

Acceptance Criteria

• When automatic restart, startup recovery or lifecycle recovery finds a backend mismatch, the sandbox's Ready condition shows a specific reason and the existing message from validate_isolation_backend_binding.
• The restart scan stops retrying that sandbox until its backend selection matches again or the user acts on it.
• A sandbox rejected by lifecycle recovery no longer ends with ImagePreparationTimedOut.
• After the operator restores the original selection, the documented recovery path (for example openshell sandbox start) brings the sandbox back.

Reproduction Steps

1. Start a gateway with the built-in isolation backend.
2. Run openshell sandbox create ... with a main process that exits on demand, and wait for Ready.
3. Stop the gateway. Register a second backend under [[openshell.supervisor.isolation_backends]] named other, and set [openshell.drivers.<name>].isolation_backend = "other". The examples/isolation-backend-v8 README shows the format.
4. Start the gateway. Make the sandbox's main process exit so an automatic restart is scheduled.
5. Run openshell sandbox get <name>.
   • Observed: the phase stays Starting with a restart reason such as MainProcessRestartScheduled, and the gateway logs a warning every 500 ms.
   • Expected: an error reason that names the mismatch.

Suggested UX (if applicable)

$ openshell sandbox get demo
Phase:  Error
Reason: IsolationBackendMismatch
        sandbox isolation backend 'openshell-sandbox' does not match gateway selection 'other';
        restore the gateway selection or create a new sandbox

Environment

• OpenShell: main at 7417f7538
• OS: platform-independent (reviewed on a macOS host)
• OpenShell deployment mode and runtime: gateway with [openshell.drivers.<name>].isolation_backend changed after sandboxes exist

Logs