Skip to content

fix(server): scope the SSH identity lock per sandbox and bound data-pool lock holders #4686

Open

Description

@EmilienM

User Story

I run multi-replica OpenShell gateways with PostgreSQL. Creating several sandboxes at once shouldn't queue them on one fleet-wide lock, and a small data pool shouldn't let a few lock holders stall every database request on a replica.

Problem Statement

The SSH host identity lock from #4094 is fleet-wide: one advisory key (SSH_IDENTITY_LOCK_KEY) and a process-wide mutex in SshIdentityStore (crates/openshell-server/src/ssh_identity.rs), both held across credential-driver calls. Sandbox create waits for it while it holds its sandbox mutation guard, and start, restart and delete wait for it under their local sandbox lock. So a burst of creates on one replica does SSH setup one sandbox at a time, and every create in the queue keeps a lock-pool connection (4 by default, database_lock_max_connections).

That lock and the provider refresh lock (acquire_refresh_guard) both hold their advisory lock on a data-pool connection, then query the same pool. The PostgreSQL floor of 2 for database_max_connections only accounts for the SSH identity lock. A few concurrent refreshes (up to 4 per replica through REFRESH_EXECUTORS; the pending-deletion cleanup path takes no permit), or one refresh next to an SSH identity holder, can hold every data connection and then wait for one more until the acquire times out after 30 seconds. Every database request on that replica stalls meanwhile.

The default pool of 10 is safe today only because the process-wide mutex allows one identity holder per replica. A per-sandbox lock removes that cap, so these two fixes have to land together.

Impact / Why This Matters

Concurrent creates wait on each other's credential-driver calls while holding lock connections other mutations need. With a small data pool, a few refreshes can stall a whole replica for 30 seconds.

Proposed Design

  • The identity lock takes a per-sandbox advisory key and a per-sandbox local mutex. It also takes the old fleet-wide key in shared mode: replicas from earlier releases hold that key exclusively, so old and new replicas still exclude each other during a rolling upgrade.
  • Data-pool advisory-lock holders on a replica share a bound below the data pool size, so a connection always stays free for their queries. Waiting for a slot gets the same 10-second bound as the lock itself.

Acceptance Criteria

  • At the smallest accepted database_max_connections, concurrent provider refreshes and SSH identity holders can't exhaust the data pool, with a PostgreSQL test.
  • The SSH host identity lock is scoped per sandbox, and creates for different sandboxes don't wait on each other.

Alternatives Considered

  • Raising the PostgreSQL floor above the number of concurrent holders only works while something caps the holders, and the per-sandbox lock removes the cap.
  • Moving the identity lock to the lock pool: create already holds a lock connection there, so a burst of creates would wait on itself.

Part of #4307.

Activity

  1. added
    state:acceptedA maintainer decided OpenShell should pursue this issue
    and removed
    state:triage-neededOpened without agent diagnostics and needs triage
    on Oct 19, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    state:acceptedA maintainer decided OpenShell should pursue this issue

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions