Skip to content

fix: close the root container after shutdown receivers connected after setup_di - #48

Merged
lesnik512 merged 1 commit into
mainfrom
fix/close-root-after-shutdown-receivers
Oct 17, 2026
Merged

lesnik512 merged 1 commit into
mainfrom
fix/close-root-after-shutdown-receivers

Conversation

@lesnik512

Copy link
Copy Markdown
Member

setup_di connected its close receivers to worker_shutdown and worker_process_shutdown when it was called. Celery calls receivers in connection order, so a shutdown receiver connected after setup_di got ContainerClosedError, and APP finalizers had already run.

How Celery 5.6.3 dispatches signals

Signal.receivers is a list of (lookup_key, receiver) pairs. connect appends unless the key is already present, and send calls the receivers in list order (celery/utils/dispatch/signal.py, _connect_signal and _live_receivers). Without dispatch_uid the key is (id(receiver), id(sender)), so disconnect(receiver) followed by connect(receiver, weak=False) moves a receiver to the end without duplicating it. send iterates a copy built by _live_receivers, so changing another signal's list during a send is safe.

Approach

The worker_init receiver moves _close_for_worker to the end of worker_shutdown, and the worker_process_init receiver moves _close_for_process to the end of worker_process_shutdown. A receiver connected before the worker starts, which is where user code connects them, now runs before the close. Each restart is a disconnect and a connect of the same function, so the receiver count stays the same. Prefork and spawn children run worker_process_init in the child, so the move happens there too. Nothing else changed: the ownership check and manage_lifespan=False work as before.

Warm shutdown with the threads and solo pools

Running tasks finish before worker_shutdown fires, so nothing runs after the close.

  • The warm shutdown signal handler only sets state.should_stop (celery/apps/worker.py, _shutdown_handler). The consumer loop raises WorkerShutdown from state.maybe_shutdown() between events (celery/worker/loops.py, celery/worker/state.py). That happens outside the signal handler, so signal_safe = False on the threads pool does not skip the shutdown.
  • WorkController.start catches it as SystemExit and calls stop(), which runs _shutdown(warm=True) (stopping the bootsteps in reverse order, the pool included) and only then _send_worker_shutdown() (celery/worker/worker.py).
  • The threads pool's on_stop calls ThreadPoolExecutor.shutdown(), which waits for running and queued work (celery/concurrency/thread.py). The solo pool runs tasks inline in the loop, so a running task finishes before maybe_shutdown() gets a chance to raise.

I checked this with a real celery worker on both pools, using a memory broker and a 2 second task that resolves an APP dependency, with SIGTERM sent during the task. The order was task:start, task:end, user worker_shutdown receiver (live), finalized. On main the same script printed finalized before the user receiver's ContainerClosedError.

Limits

  • A receiver connected after the worker starts, for example from inside another worker_init receiver that runs after ours, still runs after the close.
  • A cold shutdown goes through WorkController.terminate(), which never sends worker_shutdown, so the root is not closed there. That was already true before this change.
  • Prefork could not be run locally (macOS, spawn). The process pair is covered in-process the same way as the repro.

Changes

  • modern_di_celery/main.py: the init receivers move their close receiver to the end of its shutdown signal.
  • tests/test_shutdown_order.py: for both signal pairs, a shutdown receiver connected after setup_di resolves an APP dependency before the finalizer runs, two restart cycles leave the receiver counts unchanged and finalize once per cycle, and manage_lifespan=False neither opens nor closes the root. The first test failed on main with ContainerClosedError. The issue's repro now prints ['user:live', 'finalized'] in all four cases.
  • README.md: say when the root closes.

Closes #47

@lesnik512
lesnik512 merged commit cfa9f9c into main Oct 17, 2026
12 checks passed
@lesnik512
lesnik512 deleted the fix/close-root-after-shutdown-receivers branch October 17, 2026 13:55
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

The root container closes before shutdown signal receivers connected after setup_di

1 participant