Skip to content

fix(docker): reconnect the Docker client after a failed boot connect (#4406) - #4555

Open
ericdelorefice wants to merge 9 commits into
Abilityai:devfrom
ericdelorefice:fix/4406-docker-client-reconnect-full
Open

ericdelorefice wants to merge 9 commits into
Abilityai:devfrom
ericdelorefice:fix/4406-docker-client-reconnect-full

Conversation

@ericdelorefice

@ericdelorefice ericdelorefice commented Nov 13, 2026 •

Copy link
Copy Markdown
Contributor

Description

What to review: services/docker_service.py, services/__init__.py and tests/unit/test_4406_docker_client_reconnect.py. The rest is the mechanical move of each caller to the accessor.

This meets acceptance items 1, 2 and 4 of #4406. Item 3 is not in it, see below.

It touches 69 files, more than the ~50 in the checklist: 19 source files, 1 doc and 49 test files. The contributing guide asks a bug fix to cover every call site, and the test changes are stub and patch updates that break if any importer is left on the old name. If you would rather have it split, I can do the accessor and docker_service first and the callers after.

Per #4406, services/docker_service.py calls docker.from_env() once at import. If Docker is not reachable then, docker_client stays None for the life of the process, and every module that reads it reports no containers until the backend restarts.

get_docker_client() in docker_service is now the one way to get the client. It returns the client when there is one. Otherwise it starts one from_env() attempt on a short lived daemon thread and returns None right away, and the attempt stores the client for the next call. So no caller waits on a connect, including the async functions that call it directly (crud, the ssh sweep, the seed services, execute_command_in_container, the websocket readers) and the sync helpers that async routes call. Attempts are spaced 1s, 2s, 4s up to 60s so the 5s poll loops do not hit a dead socket on every call. The next attempt is counted from when the failed one ended, so a slow failure (a TCP DOCKER_HOST timing out) still waits out the delay. A non blocking lock means one attempt at a time. A failed attempt goes through the existing _warn_throttled with its own slot, so it logs at most once a minute.

Every caller moved to it:

  1. The readers inside docker_service and execute_command_in_container.
  2. Every module that did from services.docker_service import docker_client (main.py, the ops, setup, system_agent and telemetry routers, agent_service crud and terminal, docker_utils, cornelius, fleet_ops, ssh, system_seed, canary/snapshot). Where the import was only unused, I dropped it (helpers, lifecycle, system_agent_service).
  3. services/__init__.py also exported docker_client. It now serves it through a module __getattr__, so services.docker_client follows a reconnect too.
  4. monitoring_service built its own client with from_env() at import, it now uses the accessor.

Return values are unchanged. docker_utils keeps its existing None guards, they check the accessor now.

Things to know:

  • The first call after Docker comes back still gets None. The call after it, usually the next poll, gets the client. That includes the monitoring health check, which used to build its own client and now reports Docker unavailable on that first check. The async helpers in docker_utils also get the client inside the executor callable, and the telemetry containers route checks it through asyncio.to_thread.
  • from_env() sets no timeout of its own, so an attempt that hangs on a TCP DOCKER_HOST holds the lock until it ends, and callers keep getting None meanwhile. None of them waits on it.
  • New code should call get_docker_client(). The services.docker_client shim follows a reconnect only on attribute reads, a from services import docker_client still binds the import time value.
  • Agent create reads the client once and passes the answer to the bug: agent creation silently reuses a pre-existing Docker volume of the same name (get-then-create, no adopt/refuse decision) #1667 leftover volume guard, so a reconnect between two reads can't skip the guard and then build. No client means 503, as before.
  • If the connect thread can't start, get_docker_client() returns None and the next attempt waits out the delay, the same as a failed connect.
  • main.py's startup agent scan still only runs if Docker answers at boot. It is not rerun when Docker comes back. That needs a hook on reconnect, I left it out.

Not in this PR: the third acceptance item, roster readers telling "Docker unreadable" apart from "no containers". agent_container_states() already does that (#2196). list_all_agents_fast() and get_agent_container() still return [] and None, changing that changes what their callers get. So this PR refs #4406 instead of closing it.

On an install that runs in demo mode on purpose, Docker still unreachable logs at INFO, at most once a minute while something polls the roster. A thread that can't start still logs a WARNING.

Related Issue

Refs #4406

Journey Impact

Journey Impact: none: backend Docker client reconnect; callers keep their return values, no journey in tests/journeys/catalog.yaml covers boot with Docker down

Type of Change

  • Bug fix (non-breaking change that fixes an issue)
  • New feature (non-breaking change that adds functionality)
  • Breaking change (fix or feature that would cause existing functionality to change)
  • Documentation update

Testing

New tests in tests/unit/test_4406_docker_client_reconnect.py, with from_env and time.monotonic patched:

The connect attempt runs inline in these tests (the thread start is patched) so each one can check the state right after, except test 7, which uses the real thread.

  1. Boot with Docker down: list_all_agents_fast() returns []. Docker comes up, the clock moves past the backoff, the next call connects and the one after lists through the new client.
  2. While Docker stays down there is no second attempt inside the delay, one attempt after it, the delay doubles and stops at 60s.
  3. Once connected, the client is reused and from_env is not called again.
  4. A failure that takes longer than the delay still waits the delay before the next attempt.
  5. A caller that finds the connect lock held gets None and does not call from_env, the next caller connects.
  6. A module outside docker_service gets the new client: boot with Docker down, Docker comes up, docker_utils.container_get() returns the container from the new client.
  7. A reconnect attempt never blocks the caller: called on the event loop, get_docker_client() returns None at once while from_env waits for a coroutine on the loop, which can only run if the attempt is on another thread. The attempt then stores the client and the next call gets it. The test joins the worker thread before it ends, so no attempt outlives it.
  8. A thread that can't start: get_docker_client() returns None, the lock is free, and the next start waits out the delay.
  9. Docker still unreachable is logged at INFO.

Create, in tests/unit/test_1484_create_agent_characterization.py (its create harness) and tests/unit/test_4406_create_reads_docker_once.py:

  • get_docker_client() answers None then the client while a leftover workspace volume exists: create returns 503 and builds nothing. Before the review fix the guard read None and skipped, and the build read the client.
  • The guard goes by the value create passes it and does no read of its own, and still returns 409 for a leftover volume when Docker is up.

test_4529_cpu_percent_of_limit.py now patches services.docker_service.get_docker_client. With check_docker_health switched back to a client bound at import, its 11 tests fail.

Existing tests: tests that patched docker_client on an importer module now patch services.docker_service.get_docker_client, and the module stubs carry get_docker_client. Six tests or fixtures set docker_client = None to mean "Docker unreachable" (in test_2196_roster_availability.py, test_2215_port_allocation.py, test_docker_service_list_fast.py and test_ent403_workspace_model.py). With the accessor that now means "try again", so on a runner where Docker is up they would connect and get a real answer. Each one also sets the next attempt to never. No assertion was changed. Tests that replace services.docker_service in sys.modules now also set it on the services package, because from services import docker_service reads the package attribute first. I checked the "Docker unreachable" fixtures with a plugin that makes from_env() return an empty daemon, like a CI runner.

Full unit lane as CI runs it, on this branch rebased onto dev at c5ba9d3 (-m "not slow" --dist loadfile -p randomly --randomly-seed=4406, 4 workers, Python 3.13): 41758 passed, 131 skipped, 8 xfailed, 49 failed. Unmodified dev with the same seed fails the same 49 tests: 39 in test_ent926_turn_tier_hook.py, 4 in test_ent1109_credential_handover.py::TestTheAsk, 4 in test_4499_vendor_egress_switch.py::TestTheServedPolicy and 2 in test_ent1094_marketplace_release.py. They come from my local setup or test order, not this change. Then I rebased once more onto dev at 00b1e5a, since #4388 also changes crud.py and test_1484_create_agent_characterization.py, and ran the 128 unit test files that touch Docker or this change: 4703 passed, 1 failed, test_ent872_fleet_reach.py::test_the_event_stream_endpoint_scopes_an_agent_key, which fails the same way on dev when test_voice_tools.py runs first in the same worker (its google module stub). tests/lint_sys_modules.py and tests/lint_root_test_placement.py both OK.

Mutation: with get_docker_client() changed to return the import time value (the behaviour on dev), the first seven tests fail on their assertions. With the attempt run in the caller instead of on its own thread, test 7 fails and the others pass. With create reading the client again after the guard, the 503 create test fails. Without the delay after a failed thread start, test 8 fails.

Checklist

  • My code follows the project's style guidelines
  • I have updated the documentation (if applicable)
  • I have not committed any sensitive data (API keys, credentials, etc.)
  • I have added appropriate logging for new functionality

@ericdelorefice
ericdelorefice marked this pull request as ready for review November 13, 2026 17:47
@vybe

vybe commented Nov 18, 2026

Copy link
Copy Markdown
Contributor

merge-train 2026-11-18: not in this batch. Validated at fe69aeac2. The reconnect accessor itself looks sound: the non-blocking lock, the backoff and the tri-state readers all hold up, and test_4406_docker_client_reconnect.py exercises the real recovery path. Three things keep it off this train:

  1. A dev test fails once this PR is merged. tests/unit/test_4529_cpu_percent_of_limit.py:118 (added by fix(monitoring): scale agent CPU alerts to the container limit #4530, after this PR's base) patches monitoring_service.docker_client, which this PR removes. On dev plus this PR, 11 of 11 tests fail with "Expected 'get' to be called once. Called 0 times", even though git reports no conflict. Fix: patch services.docker_service.get_docker_client, as the other 47 test files do. That also pins check_docker_health. At present, switching it back to a stale import-time binding leaves every monitoring test green.
  2. Closes #4406 while acceptance item 3 is unmet. list_all_agents_fast() and get_agent_container() still can't tell "Docker unreadable" apart from "no containers". Either change the keyword to Refs #4406, or open a follow-up issue for item 3, link it, and narrow bug: the Docker client is bound once at import and never reconnects — a backend that boots before the socket is reachable shows an empty fleet until restart #4406.
  3. Check-then-act on agent create. crud._guard_leftover_workspace_volume (crud.py:623) and create_agent_internal (crud.py:4022) now call get_docker_client() separately, with slow awaits in between. Right as Docker comes back, the first call can get None, which skips the bug: agent creation silently reuses a pre-existing Docker volume of the same name (get-then-create, no adopt/refuse decision) #1667 leftover-volume check, while the second gets the client, so a leftover workspace volume is silently adopted instead of returning 409. Suggest reading the client once per create and passing it down.

Minor: if _start_docker_connect can't start its thread, get_docker_client() re-raises outside the readers' try. On the demo-mode log level you asked about, INFO seems right for an install that runs without Docker on purpose.

Once fixed, rebase onto dev and this rides the next train.

@vybe vybe added the status-needs-fix PR has an unaddressed review/validation finding; cleared by the author's next push (#2815) label Nov 18, 2026
…boot connect (Abilityai#4406)

docker_service bound its client once at import. When docker.from_env() failed
there, docker_client stayed None and every reader in the module reported no
containers until the backend restarted.

get_docker_client() retries from_env() with backoff (1s doubling to 60s) and the
module's readers go through it. Modules that import docker_client by value keep
the import-time object and are left for follow-up slices.

Tests that model an unreachable daemon by setting docker_client = None now also
pin the next reconnect attempt to never, so on a runner with Docker they keep
testing the unreachable case instead of connecting.
…e failure (Abilityai#4406)

A non-blocking lock keeps concurrent callers from each connecting; a caller that
finds a connect in progress gets None instead of waiting. The next attempt is
scheduled from when the failed attempt ended, so a slow failure (a TCP
DOCKER_HOST timing out) still waits out the delay.
…Abilityai#4406)

Modules that did `from services.docker_service import docker_client` kept the
value from import time, so after a boot with Docker down they stayed on None even
once docker_service reconnected. They now call docker_service.get_docker_client()
at the point of use. monitoring_service drops its own import-time from_env()
client and uses the same accessor. services.docker_client still works and is
read at access time; get_docker_client is exported too.

main.py's startup agent scan still runs only if Docker answers at boot; it is not
rerun on reconnect.

Tests that patched docker_client on an importer now patch
services.docker_service.get_docker_client, and module stubs carry
get_docker_client. New test: a docker_utils call recovers after Docker comes up.

Closes Abilityai#4406
…ce in test stubs (Abilityai#4406)

The async docker_utils helpers called get_docker_client() on the event loop, so a reconnect attempt (docker.from_env) could block it. Acquisition now happens in the executor callable. The telemetry containers route guard uses asyncio.to_thread.

Tests that replace services.docker_service also set the services package attribute, which 'from services import docker_service' reads first.
)

get_docker_client() is called directly from async functions (crud, ssh sweep,
seed services, execute_command_in_container, websocket readers) and from sync
helpers that async routes call. A reconnect attempt made in the caller could
block the event loop for a TCP DOCKER_HOST timeout. The attempt now runs on a
short lived daemon thread; the caller gets None at once and the next call gets
the client.
…i#4406)

- test_4529_cpu_percent_of_limit: patch docker_service.get_docker_client
  instead of the removed monitoring_service.docker_client (11 of 11 failed
  on dev plus this branch).
- crud: create reads the Docker client once and passes it to the Abilityai#1667
  leftover-volume guard, so the guard and the container build agree. Two
  reads could split while Docker reconnects: the guard skipped on None and
  the build then adopted a leftover workspace volume instead of a 409.
- docker_service: a reconnect thread that cannot start returns None instead
  of raising into the readers.
@ericdelorefice
ericdelorefice force-pushed the fix/4406-docker-client-reconnect-full branch from fe69aea to 30e0600 Compare November 19, 2026 17:02
@ericdelorefice

Copy link
Copy Markdown
Contributor Author

Thanks for catching the test_4529 break. All three are fixed and rebased onto dev: create reads the client once for _guard_leftover_workspace_volume and the build, and Refs #4406 keeps the issue open for item 3, or I can open a follow up issue. get_docker_client() returns None and backs off when its thread can't start, and demo mode logs at INFO.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

status-needs-fix PR has an unaddressed review/validation finding; cleared by the author's next push (#2815)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants