Skip to content

fix(process-pool): bound shutdown latency#469

Open
dexhunter wants to merge 1 commit into
algorithmicsuperintelligence:mainfrom
dexhunter:dex/bound-process-pool-shutdown
Open

fix(process-pool): bound shutdown latency#469
dexhunter wants to merge 1 commit into
algorithmicsuperintelligence:mainfrom
dexhunter:dex/bound-process-pool-shutdown

Conversation

@dexhunter

Copy link
Copy Markdown

Fixes #297.

Summary

  • prevent ProcessParallelController.stop() from waiting indefinitely for running process-pool work
  • use the public worker-termination API on Python 3.14 and a bounded compatibility path on Python 3.10-3.13
  • cancel queued work, terminate running workers, and escalate survivors to kill() after a bounded wait
  • add regression coverage for stuck workers, cleanup idempotence, and kill escalation

Why

ProcessPoolExecutor future.cancel() cannot cancel a task that is already running. The existing shutdown(wait=True) therefore waits for that task to finish, which matches the 27+ second and multi-hour shutdown hangs reported in #297.

The new path captures only this executor's workers, requests immediate shutdown, waits for those processes to exit, and escalates only workers that survive termination. Unrelated processes are not touched.

Performance

A strict scaled reproduction measures process_pool_shutdown_ms across multiple task durations with correctness gates for completed results, queued/running futures, worker cleanup, repeated stop(), post-return side effects, and unrelated processes.

Variant Metric
Current main, Python 3.10 median 1496.85 ms
Cleaned patch, Python 3.10 median 3.62 ms
Cleaned patch, Python 3.14 median 2.31 ms

That is about a 99.76% reduction (about 414x) on the minimum supported Python version. A separate 30-second worker stopped in 3.48 ms with no live worker left behind. Idle-pool shutdown remained effectively unchanged (3.40 ms before, 3.58 ms after).

Supplementary autoresearch: 20-step public dashboard.

Validation

  • Python 3.10: python -m unittest discover tests (392 passed)
  • Python 3.14: python -m unittest discover tests (392 passed)
  • Python 3.10: focused process-parallel tests (8 passed)
  • Python 3.14: focused process-parallel tests (8 passed)
  • strict shutdown evaluator on Python 3.10 and Python 3.14
  • Black on both touched files
  • git diff --check

@CLAassistant

Copy link
Copy Markdown

CLA assistant check
Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.
You have signed the CLA already but the status is still pending? Let us recheck it.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Evaluator timeouts do not seem to work, system still hangs

2 participants