Repository navigation
fix(docker): reconnect the Docker client after a failed boot connect (#4406) - #4555
ericdelorefice wants to merge 9 commits into
Conversation
|
merge-train 2026-11-18: not in this batch. Validated at
Minor: if Once fixed, rebase onto |
…boot connect (Abilityai#4406) docker_service bound its client once at import. When docker.from_env() failed there, docker_client stayed None and every reader in the module reported no containers until the backend restarted. get_docker_client() retries from_env() with backoff (1s doubling to 60s) and the module's readers go through it. Modules that import docker_client by value keep the import-time object and are left for follow-up slices. Tests that model an unreachable daemon by setting docker_client = None now also pin the next reconnect attempt to never, so on a runner with Docker they keep testing the unreachable case instead of connecting.
…e failure (Abilityai#4406) A non-blocking lock keeps concurrent callers from each connecting; a caller that finds a connect in progress gets None instead of waiting. The next attempt is scheduled from when the failed attempt ended, so a slow failure (a TCP DOCKER_HOST timing out) still waits out the delay.
…Abilityai#4406) Modules that did `from services.docker_service import docker_client` kept the value from import time, so after a boot with Docker down they stayed on None even once docker_service reconnected. They now call docker_service.get_docker_client() at the point of use. monitoring_service drops its own import-time from_env() client and uses the same accessor. services.docker_client still works and is read at access time; get_docker_client is exported too. main.py's startup agent scan still runs only if Docker answers at boot; it is not rerun on reconnect. Tests that patched docker_client on an importer now patch services.docker_service.get_docker_client, and module stubs carry get_docker_client. New test: a docker_utils call recovers after Docker comes up. Closes Abilityai#4406
…ce in test stubs (Abilityai#4406) The async docker_utils helpers called get_docker_client() on the event loop, so a reconnect attempt (docker.from_env) could block it. Acquisition now happens in the executor callable. The telemetry containers route guard uses asyncio.to_thread. Tests that replace services.docker_service also set the services package attribute, which 'from services import docker_service' reads first.
) get_docker_client() is called directly from async functions (crud, ssh sweep, seed services, execute_command_in_container, websocket readers) and from sync helpers that async routes call. A reconnect attempt made in the caller could block the event loop for a TCP DOCKER_HOST timeout. The attempt now runs on a short lived daemon thread; the caller gets None at once and the next call gets the client.
…i#4406) - test_4529_cpu_percent_of_limit: patch docker_service.get_docker_client instead of the removed monitoring_service.docker_client (11 of 11 failed on dev plus this branch). - crud: create reads the Docker client once and passes it to the Abilityai#1667 leftover-volume guard, so the guard and the container build agree. Two reads could split while Docker reconnects: the guard skipped on None and the build then adopted a leftover workspace volume instead of a 409. - docker_service: a reconnect thread that cannot start returns None instead of raising into the readers.
…r a failed thread start
fe69aea to
30e0600
Compare
|
Thanks for catching the test_4529 break. All three are fixed and rebased onto |
Description
What to review:
services/docker_service.py,services/__init__.pyandtests/unit/test_4406_docker_client_reconnect.py. The rest is the mechanical move of each caller to the accessor.This meets acceptance items 1, 2 and 4 of #4406. Item 3 is not in it, see below.
It touches 69 files, more than the ~50 in the checklist: 19 source files, 1 doc and 49 test files. The contributing guide asks a bug fix to cover every call site, and the test changes are stub and patch updates that break if any importer is left on the old name. If you would rather have it split, I can do the accessor and docker_service first and the callers after.
Per #4406,
services/docker_service.pycallsdocker.from_env()once at import. If Docker is not reachable then,docker_clientstaysNonefor the life of the process, and every module that reads it reports no containers until the backend restarts.get_docker_client()indocker_serviceis now the one way to get the client. It returns the client when there is one. Otherwise it starts onefrom_env()attempt on a short lived daemon thread and returnsNoneright away, and the attempt stores the client for the next call. So no caller waits on a connect, including the async functions that call it directly (crud, the ssh sweep, the seed services,execute_command_in_container, the websocket readers) and the sync helpers that async routes call. Attempts are spaced 1s, 2s, 4s up to 60s so the 5s poll loops do not hit a dead socket on every call. The next attempt is counted from when the failed one ended, so a slow failure (a TCPDOCKER_HOSTtiming out) still waits out the delay. A non blocking lock means one attempt at a time. A failed attempt goes through the existing_warn_throttledwith its own slot, so it logs at most once a minute.Every caller moved to it:
docker_serviceandexecute_command_in_container.from services.docker_service import docker_client(main.py, the ops, setup, system_agent and telemetry routers, agent_service crud and terminal, docker_utils, cornelius, fleet_ops, ssh, system_seed, canary/snapshot). Where the import was only unused, I dropped it (helpers, lifecycle, system_agent_service).services/__init__.pyalso exporteddocker_client. It now serves it through a module__getattr__, soservices.docker_clientfollows a reconnect too.monitoring_servicebuilt its own client withfrom_env()at import, it now uses the accessor.Return values are unchanged. docker_utils keeps its existing
Noneguards, they check the accessor now.Things to know:
None. The call after it, usually the next poll, gets the client. That includes the monitoring health check, which used to build its own client and now reports Docker unavailable on that first check. The async helpers in docker_utils also get the client inside the executor callable, and the telemetry containers route checks it throughasyncio.to_thread.from_env()sets no timeout of its own, so an attempt that hangs on a TCPDOCKER_HOSTholds the lock until it ends, and callers keep gettingNonemeanwhile. None of them waits on it.get_docker_client(). Theservices.docker_clientshim follows a reconnect only on attribute reads, afrom services import docker_clientstill binds the import time value.get_docker_client()returnsNoneand the next attempt waits out the delay, the same as a failed connect.Not in this PR: the third acceptance item, roster readers telling "Docker unreadable" apart from "no containers".
agent_container_states()already does that (#2196).list_all_agents_fast()andget_agent_container()still return[]andNone, changing that changes what their callers get. So this PR refs #4406 instead of closing it.On an install that runs in demo mode on purpose,
Docker still unreachablelogs at INFO, at most once a minute while something polls the roster. A thread that can't start still logs a WARNING.Related Issue
Refs #4406
Journey Impact
Journey Impact: none: backend Docker client reconnect; callers keep their return values, no journey in tests/journeys/catalog.yaml covers boot with Docker down
Type of Change
Testing
module under test unless its live consumer is CI itself (a workflow, a vendored copy, a
route table), and no bound check at the one value where both bounds coincide (bug(process): tests that assert source TEXT instead of executing the path — three green-CI ejections in one train, third run running #2829)
New tests in
tests/unit/test_4406_docker_client_reconnect.py, withfrom_envandtime.monotonicpatched:The connect attempt runs inline in these tests (the thread start is patched) so each one can check the state right after, except test 7, which uses the real thread.
list_all_agents_fast()returns[]. Docker comes up, the clock moves past the backoff, the next call connects and the one after lists through the new client.from_envis not called again.Noneand does not callfrom_env, the next caller connects.docker_utils.container_get()returns the container from the new client.get_docker_client()returnsNoneat once whilefrom_envwaits for a coroutine on the loop, which can only run if the attempt is on another thread. The attempt then stores the client and the next call gets it. The test joins the worker thread before it ends, so no attempt outlives it.get_docker_client()returnsNone, the lock is free, and the next start waits out the delay.Docker still unreachableis logged at INFO.Create, in
tests/unit/test_1484_create_agent_characterization.py(its create harness) andtests/unit/test_4406_create_reads_docker_once.py:get_docker_client()answersNonethen the client while a leftover workspace volume exists: create returns 503 and builds nothing. Before the review fix the guard readNoneand skipped, and the build read the client.test_4529_cpu_percent_of_limit.pynow patchesservices.docker_service.get_docker_client. Withcheck_docker_healthswitched back to a client bound at import, its 11 tests fail.Existing tests: tests that patched
docker_clienton an importer module now patchservices.docker_service.get_docker_client, and the module stubs carryget_docker_client. Six tests or fixtures setdocker_client = Noneto mean "Docker unreachable" (intest_2196_roster_availability.py,test_2215_port_allocation.py,test_docker_service_list_fast.pyandtest_ent403_workspace_model.py). With the accessor that now means "try again", so on a runner where Docker is up they would connect and get a real answer. Each one also sets the next attempt to never. No assertion was changed. Tests that replaceservices.docker_serviceinsys.modulesnow also set it on theservicespackage, becausefrom services import docker_servicereads the package attribute first. I checked the "Docker unreachable" fixtures with a plugin that makesfrom_env()return an empty daemon, like a CI runner.Full unit lane as CI runs it, on this branch rebased onto
devat c5ba9d3 (-m "not slow" --dist loadfile -p randomly --randomly-seed=4406, 4 workers, Python 3.13): 41758 passed, 131 skipped, 8 xfailed, 49 failed. Unmodifieddevwith the same seed fails the same 49 tests: 39 intest_ent926_turn_tier_hook.py, 4 intest_ent1109_credential_handover.py::TestTheAsk, 4 intest_4499_vendor_egress_switch.py::TestTheServedPolicyand 2 intest_ent1094_marketplace_release.py. They come from my local setup or test order, not this change. Then I rebased once more ontodevat 00b1e5a, since #4388 also changescrud.pyandtest_1484_create_agent_characterization.py, and ran the 128 unit test files that touch Docker or this change: 4703 passed, 1 failed,test_ent872_fleet_reach.py::test_the_event_stream_endpoint_scopes_an_agent_key, which fails the same way ondevwhentest_voice_tools.pyruns first in the same worker (itsgooglemodule stub).tests/lint_sys_modules.pyandtests/lint_root_test_placement.pyboth OK.Mutation: with
get_docker_client()changed to return the import time value (the behaviour ondev), the first seven tests fail on their assertions. With the attempt run in the caller instead of on its own thread, test 7 fails and the others pass. With create reading the client again after the guard, the 503 create test fails. Without the delay after a failed thread start, test 8 fails.Checklist