Skip to content

CI failures with AssertionError: Regex pattern did not match. #9140

Closed
#9141
Closed
CI failures with AssertionError: Regex pattern did not match.#9140
#9141
Labels
flaky testIntermittent failures on CI.

Description

@jacobtomlinson
jacobtomlinson
opened on Nov 11, 2025

In #9139 I see a lot of CI failures all with the error AssertionError: Regex pattern did not match., not the same test every time though.

FAILED distributed/tests/test_scheduler.py::test_restart_nanny_timeout_exceeded - AssertionError: Regex pattern did not match.
 Regex: '2/2 nanny worker\\(s\\) did not shut down within 1s'
 Input: '[Errno None] None'
FAILED distributed/tests/test_scheduler.py::test_restart_not_all_workers_return - AssertionError: Regex pattern did not match.
 Regex: 'Waited for 2 worker'
 Input: '[Errno None] None'
FAILED distributed/tests/test_scheduler.py::test_restart_some_nannies_some_not - AssertionError: Regex pattern did not match.
 Regex: 'The 1 worker\\(s\\) not using Nannies'
 Input: '[Errno None] None'

Activity

  1. jacobtomlinson
    added
    flaky testIntermittent failures on CI.
    on Nov 11, 2025
  2. jacobtomlinson
    added a commit that references this issue on Nov 11, 2025

    Run CI to investigate dask#9140

    7540f28

  3. jacobtomlinson commented on Nov 11, 2025

    @jacobtomlinson
    jacobtomlinson
    on Nov 11, 2025
    MemberAuthor

    The last commit to main was the 2025.11.0 release. It looks like these errors happened but only for the macOS Python 3.12 runs. However in #9141 it's happening reasonably consistently across all operating systems and Python versions.

  4. jacobtomlinson commented on Nov 11, 2025

    @jacobtomlinson
    jacobtomlinson
    on Nov 11, 2025
    MemberAuthor

    I am able to reproduce this locally.

    Local environment 
    name: dask
channels:
  - conda-forge
dependencies:
  - bzip2=1.0.8=hd037594_8
  - ca-certificates=2025.10.5=hbd8a1cb_0
  - libexpat=2.7.1=hec049ff_0
  - libffi=3.5.2=he5f378a_0
  - liblzma=5.8.1=h39f12f2_2
  - libmpdec=4.0.0=h5505292_0
  - libsqlite=3.51.0=h9a5124b_0
  - libzlib=1.3.1=h8359307_2
  - ncurses=6.5=h5e97a16_3
  - openssl=3.5.4=h5503f6c_0
  - pip=25.2=pyh145f28c_0
  - python=3.13.9=hfc2f54d_101_cp313
  - python_abi=3.13=8_cp313
  - readline=8.2=h1d1bf99_2
  - tk=8.6.13=h892fb3f_2
  - pip:
      - bokeh==3.8.1
      - cfgv==3.4.0
      - click==8.3.0
      - cloudpickle==3.1.2
      - contourpy==1.3.3
      - coverage==7.11.3
      - dask==2025.11.1.dev3+ga66a89731
      - distlib==0.4.0
      - distributed==2025.11.0
      - execnet==2.1.1
      - filelock==3.20.0
      - fsspec==2025.10.0
      - hypothesis==6.147.0
      - identify==2.6.15
      - iniconfig==2.3.0
      - jinja2==3.1.6
      - locket==1.0.0
      - lz4==4.4.5
      - markupsafe==3.0.3
      - msgpack==1.1.2
      - narwhals==2.10.2
      - nodeenv==1.9.1
      - numpy==2.3.4
      - packaging==25.0
      - pandas==2.3.3
      - partd==1.4.2
      - pillow==12.0.0
      - platformdirs==4.5.0
      - pluggy==1.6.0
      - pre-commit==4.4.0
      - psutil==7.1.3
      - pyarrow==22.0.0
      - pygments==2.19.2
      - pytest==8.4.2
      - pytest-cov==7.0.0
      - pytest-mock==3.15.1
      - pytest-rerunfailures==16.1
      - pytest-timeout==2.4.0
      - pytest-xdist==3.8.0
      - python-dateutil==2.9.0.post0
      - pytz==2025.2
      - pyyaml==6.0.3
      - six==1.17.0
      - sortedcontainers==2.4.0
      - tblib==3.2.1
      - toolz==1.1.0
      - tornado==6.5.2
      - tzdata==2025.2
      - urllib3==2.5.0
      - virtualenv==20.35.4
      - xyzservices==2025.10.0
      - zict==3.0.0
prefix: /Users/jtomlinson/miniconda3/envs/dask

    The assertion failure is very consistent, it happens when a TimeoutError is raised when restarting workers. All of the tests are failing to match an error message.

    distributed/distributed/scheduler.py

    Line 6709 in fde9e5f

    raise TimeoutError(msg)

    It looks like the exception is being raised, but when comparing the regex isn't matching due to the message being set to None.

    Traceback (most recent call last):
  File "/Users/jtomlinson/Projects/dask/distributed/distributed/tests/test_scheduler.py", line 1162, in test_restart_not_all_workers_return
    await c.restart(timeout="1s")
  File "/Users/jtomlinson/Projects/dask/distributed/distributed/client.py", line 3824, in _restart
    await self.scheduler.restart(
    ...<3 lines>...
    )
  File "/Users/jtomlinson/Projects/dask/distributed/distributed/core.py", line 1259, in send_recv_from_rpc
    return await send_recv(comm=comm, op=key, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/jtomlinson/Projects/dask/distributed/distributed/core.py", line 1043, in send_recv
    raise exc.with_traceback(tb)
  File "/Users/jtomlinson/Projects/dask/distributed/distributed/core.py", line 834, in _handle_comm
    result = await result
    
  File "/Users/jtomlinson/Projects/dask/distributed/distributed/utils.py", line 825, in wrapper
    return await func(*args, **kwargs)
    ^^^
  File "/Users/jtomlinson/Projects/dask/distributed/distributed/scheduler.py", line 6526, in restart
    await self.restart_workers(
    ^^^^^^^^^^^^^^^
  File "/Users/jtomlinson/Projects/dask/distributed/distributed/utils.py", line 825, in wrapper
    return await func(*args, **kwargs)
    ^^^
  File "/Users/jtomlinson/Projects/dask/distributed/distributed/scheduler.py", line 6710, in restart_workers
    raise TimeoutError(msg)
    ^^^^^^^^^^^
TimeoutError: [Errno None] None

  5. jacobtomlinson commented on Nov 11, 2025

    @jacobtomlinson
    jacobtomlinson
    on Nov 11, 2025
    MemberAuthor

    Running the test locally I can clearly see the exception being raised with the correct message which should match the regex, but by the time it gets to the test the message is wrong.

    Exception logged to stderr

    TimeoutError: Waited for 2 worker(s) to reconnect after restarting but, after 1s, 2 have not returned. Consider a longer timeout, or `wait_for_workers=False`. The 2 worker(s) not using Nannies were just shut down instead of restarted (restart is only possible with Nannies). If your deployment system does not automatically re-launch terminated processes, then those workers will never come back, and `Client.restart` will always time out. Do not use `Client.restart` in that case.

    Test output

    AssertionError: Regex pattern did not match.
 Regex: 'Waited for 2 worker'
 Input: '[Errno None] None'
    Full test output 
    pytest -s --tb=native --runslow distributed/tests/test_scheduler.py::test_restart_not_all_workers_return
============================================================ test session starts =============================================================
platform darwin -- Python 3.13.9, pytest-8.4.2, pluggy-1.6.0 -- /Users/jtomlinson/miniconda3/envs/dask/bin/python3.13
cachedir: .pytest_cache
hypothesis profile 'default'
rootdir: /Users/jtomlinson/Projects/dask/distributed
configfile: pyproject.toml
plugins: mock-3.15.1, hypothesis-6.147.0, xdist-3.8.0, timeout-2.4.0, rerunfailures-16.1, cov-7.0.0
timeout: 300.0s
timeout method: thread
timeout func_only: False
collected 1 item                                                                                                                             

distributed/tests/test_scheduler.py::test_restart_not_all_workers_return 2025-11-11 17:11:14,335 - distributed.http.proxy - INFO - To route to workers diagnostics web server please install jupyter-server-proxy: python -m pip install jupyter-server-proxy
2025-11-11 17:11:14,339 - distributed.scheduler - INFO - State start
2025-11-11 17:11:14,345 - distributed.scheduler - INFO -   Scheduler at:     tcp://127.0.0.1:59121
2025-11-11 17:11:14,345 - distributed.scheduler - INFO -   dashboard at:  http://127.0.0.1:59120/status
2025-11-11 17:11:14,345 - distributed.scheduler - INFO - Registering Worker plugin shuffle
2025-11-11 17:11:14,361 - distributed.worker - INFO -       Start worker at:      tcp://127.0.0.1:59122
2025-11-11 17:11:14,361 - distributed.worker - INFO -          Listening to:      tcp://127.0.0.1:59122
2025-11-11 17:11:14,361 - distributed.worker - INFO -           Worker name:                          0
2025-11-11 17:11:14,361 - distributed.worker - INFO -          dashboard at:            127.0.0.1:59123
2025-11-11 17:11:14,361 - distributed.worker - INFO - Waiting to connect to:      tcp://127.0.0.1:59121
2025-11-11 17:11:14,362 - distributed.worker - INFO - -------------------------------------------------
2025-11-11 17:11:14,362 - distributed.worker - INFO -               Threads:                          1
2025-11-11 17:11:14,362 - distributed.worker - INFO -                Memory:                  36.00 GiB
2025-11-11 17:11:14,362 - distributed.worker - INFO -       Local Directory: /var/folders/xw/bnx6v3313s5gh8bnx5wz806r0000gp/T/dask-scratch-space/worker-pv2sco4h
2025-11-11 17:11:14,362 - distributed.worker - INFO - -------------------------------------------------
2025-11-11 17:11:14,362 - distributed.worker - INFO -       Start worker at:      tcp://127.0.0.1:59125
2025-11-11 17:11:14,362 - distributed.worker - INFO -          Listening to:      tcp://127.0.0.1:59125
2025-11-11 17:11:14,362 - distributed.worker - INFO -           Worker name:                          1
2025-11-11 17:11:14,362 - distributed.worker - INFO -          dashboard at:            127.0.0.1:59126
2025-11-11 17:11:14,362 - distributed.worker - INFO - Waiting to connect to:      tcp://127.0.0.1:59121
2025-11-11 17:11:14,362 - distributed.worker - INFO - -------------------------------------------------
2025-11-11 17:11:14,362 - distributed.worker - INFO -               Threads:                          1
2025-11-11 17:11:14,362 - distributed.worker - INFO -                Memory:                  36.00 GiB
2025-11-11 17:11:14,362 - distributed.worker - INFO -       Local Directory: /var/folders/xw/bnx6v3313s5gh8bnx5wz806r0000gp/T/dask-scratch-space/worker-_l8m2ks4
2025-11-11 17:11:14,362 - distributed.worker - INFO - -------------------------------------------------
2025-11-11 17:11:14,388 - distributed.scheduler - INFO - Register worker addr: tcp://127.0.0.1:59122 name: 0
2025-11-11 17:11:14,388 - distributed.scheduler - INFO - Starting worker compute stream, tcp://127.0.0.1:59122
2025-11-11 17:11:14,388 - distributed.core - INFO - Starting established connection to tcp://127.0.0.1:59124
2025-11-11 17:11:14,389 - distributed.scheduler - INFO - Register worker addr: tcp://127.0.0.1:59125 name: 1
2025-11-11 17:11:14,389 - distributed.scheduler - INFO - Starting worker compute stream, tcp://127.0.0.1:59125
2025-11-11 17:11:14,389 - distributed.core - INFO - Starting established connection to tcp://127.0.0.1:59127
2025-11-11 17:11:14,389 - distributed.worker - INFO - Starting Worker plugin shuffle
2025-11-11 17:11:14,389 - distributed.worker - INFO -         Registered to:      tcp://127.0.0.1:59121
2025-11-11 17:11:14,389 - distributed.worker - INFO - -------------------------------------------------
2025-11-11 17:11:14,390 - distributed.worker - INFO - Starting Worker plugin shuffle
2025-11-11 17:11:14,390 - distributed.core - INFO - Starting established connection to tcp://127.0.0.1:59121
2025-11-11 17:11:14,390 - distributed.worker - INFO -         Registered to:      tcp://127.0.0.1:59121
2025-11-11 17:11:14,390 - distributed.worker - INFO - -------------------------------------------------
2025-11-11 17:11:14,390 - distributed.core - INFO - Starting established connection to tcp://127.0.0.1:59121
2025-11-11 17:11:14,420 - distributed.scheduler - INFO - Receive client connection: Client-6db6f234-bf21-11f0-a9ea-b4e9b8685d00
2025-11-11 17:11:14,420 - distributed.core - INFO - Starting established connection to tcp://127.0.0.1:59128
2025-11-11 17:11:14,421 - distributed.scheduler - INFO - Restarting workers and releasing all keys (stimulus_id='client-restart-1762881074.4214919')
2025-11-11 17:11:14,421 - distributed.scheduler - INFO - Restarting all workers (stimulus_id='client-restart-1762881074.4214919'
2025-11-11 17:11:14,421 - distributed.scheduler - WARNING - Workers ['tcp://127.0.0.1:59122', 'tcp://127.0.0.1:59125'] do not use a nanny and will be terminated without restarting them
2025-11-11 17:11:14,421 - distributed.scheduler - INFO - Remove worker addr: tcp://127.0.0.1:59122 name: 0 (stimulus_id='client-restart-1762881074.4214919')
2025-11-11 17:11:14,422 - distributed.scheduler - INFO - Remove worker addr: tcp://127.0.0.1:59125 name: 1 (stimulus_id='client-restart-1762881074.4214919')
2025-11-11 17:11:14,422 - distributed.scheduler - INFO - Lost all workers
2025-11-11 17:11:14,422 - distributed.worker - INFO - Stopping worker at tcp://127.0.0.1:59122. Reason: scheduler-remove-worker
2025-11-11 17:11:14,422 - distributed.worker - INFO - Stopping worker at tcp://127.0.0.1:59125. Reason: scheduler-remove-worker
2025-11-11 17:11:14,422 - distributed.worker - INFO - Removing Worker plugin shuffle
2025-11-11 17:11:14,422 - distributed.worker - INFO - Removing Worker plugin shuffle
2025-11-11 17:11:14,424 - distributed.core - INFO - Received 'close-stream' from tcp://127.0.0.1:59124; closing.
2025-11-11 17:11:14,424 - distributed.core - INFO - Received 'close-stream' from tcp://127.0.0.1:59127; closing.
2025-11-11 17:11:14,429 - distributed.core - INFO - Connection to tcp://127.0.0.1:59121 has been closed.
2025-11-11 17:11:14,433 - distributed.core - INFO - Connection to tcp://127.0.0.1:59121 has been closed.
2025-11-11 17:11:15,460 - distributed.scheduler - ERROR - Waited for 2 worker(s) to reconnect after restarting but, after 1s, 2 have not returned. Consider a longer timeout, or `wait_for_workers=False`. The 2 worker(s) not using Nannies were just shut down instead of restarted (restart is only possible with Nannies). If your deployment system does not automatically re-launch terminated processes, then those workers will never come back, and `Client.restart` will always time out. Do not use `Client.restart` in that case.
Traceback (most recent call last):
  File "/Users/jtomlinson/Projects/dask/distributed/distributed/utils.py", line 825, in wrapper
    return await func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/jtomlinson/Projects/dask/distributed/distributed/scheduler.py", line 6709, in restart_workers
    raise TimeoutError(msg)
TimeoutError: Waited for 2 worker(s) to reconnect after restarting but, after 1s, 2 have not returned. Consider a longer timeout, or `wait_for_workers=False`. The 2 worker(s) not using Nannies were just shut down instead of restarted (restart is only possible with Nannies). If your deployment system does not automatically re-launch terminated processes, then those workers will never come back, and `Client.restart` will always time out. Do not use `Client.restart` in that case.
2025-11-11 17:11:15,463 - distributed.scheduler - ERROR - Waited for 2 worker(s) to reconnect after restarting but, after 1s, 2 have not returned. Consider a longer timeout, or `wait_for_workers=False`. The 2 worker(s) not using Nannies were just shut down instead of restarted (restart is only possible with Nannies). If your deployment system does not automatically re-launch terminated processes, then those workers will never come back, and `Client.restart` will always time out. Do not use `Client.restart` in that case.
Traceback (most recent call last):
  File "/Users/jtomlinson/Projects/dask/distributed/distributed/utils.py", line 825, in wrapper
    return await func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/jtomlinson/Projects/dask/distributed/distributed/scheduler.py", line 6526, in restart
    await self.restart_workers(
    ...<4 lines>...
    )
  File "/Users/jtomlinson/Projects/dask/distributed/distributed/utils.py", line 825, in wrapper
    return await func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/jtomlinson/Projects/dask/distributed/distributed/scheduler.py", line 6709, in restart_workers
    raise TimeoutError(msg)
TimeoutError: Waited for 2 worker(s) to reconnect after restarting but, after 1s, 2 have not returned. Consider a longer timeout, or `wait_for_workers=False`. The 2 worker(s) not using Nannies were just shut down instead of restarted (restart is only possible with Nannies). If your deployment system does not automatically re-launch terminated processes, then those workers will never come back, and `Client.restart` will always time out. Do not use `Client.restart` in that case.
2025-11-11 17:11:15,464 - distributed.core - ERROR - Exception while handling op restart
Traceback (most recent call last):
  File "/Users/jtomlinson/Projects/dask/distributed/distributed/core.py", line 834, in _handle_comm
    result = await result
             ^^^^^^^^^^^^
  File "/Users/jtomlinson/Projects/dask/distributed/distributed/utils.py", line 825, in wrapper
    return await func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/jtomlinson/Projects/dask/distributed/distributed/scheduler.py", line 6526, in restart
    await self.restart_workers(
    ...<4 lines>...
    )
  File "/Users/jtomlinson/Projects/dask/distributed/distributed/utils.py", line 825, in wrapper
    return await func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/jtomlinson/Projects/dask/distributed/distributed/scheduler.py", line 6709, in restart_workers
    raise TimeoutError(msg)
TimeoutError: Waited for 2 worker(s) to reconnect after restarting but, after 1s, 2 have not returned. Consider a longer timeout, or `wait_for_workers=False`. The 2 worker(s) not using Nannies were just shut down instead of restarted (restart is only possible with Nannies). If your deployment system does not automatically re-launch terminated processes, then those workers will never come back, and `Client.restart` will always time out. Do not use `Client.restart` in that case.
2025-11-11 17:11:15,509 - distributed.scheduler - INFO - Remove client Client-6db6f234-bf21-11f0-a9ea-b4e9b8685d00
2025-11-11 17:11:15,509 - distributed.core - INFO - Received 'close-stream' from tcp://127.0.0.1:59128; closing.
2025-11-11 17:11:15,509 - distributed.scheduler - INFO - Remove client Client-6db6f234-bf21-11f0-a9ea-b4e9b8685d00
2025-11-11 17:11:15,509 - distributed.scheduler - INFO - Close client connection: Client-6db6f234-bf21-11f0-a9ea-b4e9b8685d00
2025-11-11 17:11:15,510 - distributed.scheduler - INFO - Closing scheduler. Reason: unknown
2025-11-11 17:11:15,510 - distributed.scheduler - INFO - Scheduler closing all comms
FAILED

================================================================== FAILURES ==================================================================
____________________________________________________ test_restart_not_all_workers_return _____________________________________________________
Traceback (most recent call last):
  File "/Users/jtomlinson/Projects/dask/distributed/distributed/tests/test_scheduler.py", line 1162, in test_restart_not_all_workers_return
    await c.restart(timeout="1s")
  File "/Users/jtomlinson/Projects/dask/distributed/distributed/client.py", line 3824, in _restart
    await self.scheduler.restart(
    ...<3 lines>...
    )
  File "/Users/jtomlinson/Projects/dask/distributed/distributed/core.py", line 1259, in send_recv_from_rpc
    return await send_recv(comm=comm, op=key, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/jtomlinson/Projects/dask/distributed/distributed/core.py", line 1043, in send_recv
    raise exc.with_traceback(tb)
  File "/Users/jtomlinson/Projects/dask/distributed/distributed/core.py", line 834, in _handle_comm
    result = await result
    
  File "/Users/jtomlinson/Projects/dask/distributed/distributed/utils.py", line 825, in wrapper
    return await func(*args, **kwargs)
    ^^^
  File "/Users/jtomlinson/Projects/dask/distributed/distributed/scheduler.py", line 6526, in restart
    await self.restart_workers(
    ^^^^^^^^^^^^^^^
  File "/Users/jtomlinson/Projects/dask/distributed/distributed/utils.py", line 825, in wrapper
    return await func(*args, **kwargs)
    ^^^
  File "/Users/jtomlinson/Projects/dask/distributed/distributed/scheduler.py", line 6709, in restart_workers
    raise TimeoutError(msg)
    ^^^^^^^^^^^
TimeoutError: [Errno None] None

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/jtomlinson/miniconda3/envs/dask/lib/python3.13/site-packages/_pytest/runner.py", line 344, in from_call
    result: TResult | None = func()
                             ~~~~^^
  File "/Users/jtomlinson/miniconda3/envs/dask/lib/python3.13/site-packages/_pytest/runner.py", line 246, in <lambda>
    lambda: runtest_hook(item=item, **kwds), when=when, reraise=reraise
            ~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^
  File "/Users/jtomlinson/miniconda3/envs/dask/lib/python3.13/site-packages/pluggy/_hooks.py", line 512, in __call__
    return self._hookexec(self.name, self._hookimpls.copy(), kwargs, firstresult)
           ~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/jtomlinson/miniconda3/envs/dask/lib/python3.13/site-packages/pluggy/_manager.py", line 120, in _hookexec
    return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
           ~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/jtomlinson/miniconda3/envs/dask/lib/python3.13/site-packages/pluggy/_callers.py", line 167, in _multicall
    raise exception
  File "/Users/jtomlinson/miniconda3/envs/dask/lib/python3.13/site-packages/pluggy/_callers.py", line 139, in _multicall
    teardown.throw(exception)
    ~~~~~~~~~~~~~~^^^^^^^^^^^
  File "/Users/jtomlinson/miniconda3/envs/dask/lib/python3.13/site-packages/_pytest/logging.py", line 850, in pytest_runtest_call
    yield
  File "/Users/jtomlinson/miniconda3/envs/dask/lib/python3.13/site-packages/pluggy/_callers.py", line 139, in _multicall
    teardown.throw(exception)
    ~~~~~~~~~~~~~~^^^^^^^^^^^
  File "/Users/jtomlinson/miniconda3/envs/dask/lib/python3.13/site-packages/_pytest/capture.py", line 900, in pytest_runtest_call
    return (yield)
            ^^^^^
  File "/Users/jtomlinson/miniconda3/envs/dask/lib/python3.13/site-packages/pluggy/_callers.py", line 139, in _multicall
    teardown.throw(exception)
    ~~~~~~~~~~~~~~^^^^^^^^^^^
  File "/Users/jtomlinson/miniconda3/envs/dask/lib/python3.13/site-packages/pluggy/_callers.py", line 53, in run_old_style_hookwrapper
    return result.get_result()
           ~~~~~~~~~~~~~~~~~^^
  File "/Users/jtomlinson/miniconda3/envs/dask/lib/python3.13/site-packages/pluggy/_result.py", line 103, in get_result
    raise exc.with_traceback(tb)
  File "/Users/jtomlinson/miniconda3/envs/dask/lib/python3.13/site-packages/pluggy/_callers.py", line 38, in run_old_style_hookwrapper
    res = yield
          ^^^^^
  File "/Users/jtomlinson/miniconda3/envs/dask/lib/python3.13/site-packages/pluggy/_callers.py", line 139, in _multicall
    teardown.throw(exception)
    ~~~~~~~~~~~~~~^^^^^^^^^^^
  File "/Users/jtomlinson/miniconda3/envs/dask/lib/python3.13/site-packages/pluggy/_callers.py", line 53, in run_old_style_hookwrapper
    return result.get_result()
           ~~~~~~~~~~~~~~~~~^^
  File "/Users/jtomlinson/miniconda3/envs/dask/lib/python3.13/site-packages/pluggy/_result.py", line 103, in get_result
    raise exc.with_traceback(tb)
  File "/Users/jtomlinson/miniconda3/envs/dask/lib/python3.13/site-packages/pluggy/_callers.py", line 38, in run_old_style_hookwrapper
    res = yield
          ^^^^^
  File "/Users/jtomlinson/miniconda3/envs/dask/lib/python3.13/site-packages/pluggy/_callers.py", line 139, in _multicall
    teardown.throw(exception)
    ~~~~~~~~~~~~~~^^^^^^^^^^^
  File "/Users/jtomlinson/miniconda3/envs/dask/lib/python3.13/site-packages/_pytest/skipping.py", line 263, in pytest_runtest_call
    return (yield)
            ^^^^^
  File "/Users/jtomlinson/miniconda3/envs/dask/lib/python3.13/site-packages/pluggy/_callers.py", line 121, in _multicall
    res = hook_impl.function(*args)
  File "/Users/jtomlinson/miniconda3/envs/dask/lib/python3.13/site-packages/_pytest/runner.py", line 178, in pytest_runtest_call
    item.runtest()
    ~~~~~~~~~~~~^^
  File "/Users/jtomlinson/miniconda3/envs/dask/lib/python3.13/site-packages/_pytest/python.py", line 1671, in runtest
    self.ihook.pytest_pyfunc_call(pyfuncitem=self)
    ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^
  File "/Users/jtomlinson/miniconda3/envs/dask/lib/python3.13/site-packages/pluggy/_hooks.py", line 512, in __call__
    return self._hookexec(self.name, self._hookimpls.copy(), kwargs, firstresult)
           ~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/jtomlinson/miniconda3/envs/dask/lib/python3.13/site-packages/pluggy/_manager.py", line 120, in _hookexec
    return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
           ~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/jtomlinson/miniconda3/envs/dask/lib/python3.13/site-packages/pluggy/_callers.py", line 167, in _multicall
    raise exception
  File "/Users/jtomlinson/miniconda3/envs/dask/lib/python3.13/site-packages/pluggy/_callers.py", line 121, in _multicall
    res = hook_impl.function(*args)
  File "/Users/jtomlinson/miniconda3/envs/dask/lib/python3.13/site-packages/_pytest/python.py", line 157, in pytest_pyfunc_call
    result = testfunction(**testargs)
  File "/Users/jtomlinson/miniconda3/envs/dask/lib/python3.13/contextlib.py", line 85, in inner
    return func(*args, **kwds)
  File "/Users/jtomlinson/miniconda3/envs/dask/lib/python3.13/contextlib.py", line 85, in inner
    return func(*args, **kwds)
  File "/Users/jtomlinson/Projects/dask/distributed/distributed/utils_test.py", line 1092, in test_func
    return _run_and_close_tornado(async_fn_outer)
  File "/Users/jtomlinson/Projects/dask/distributed/distributed/utils_test.py", line 380, in _run_and_close_tornado
    return asyncio_run(inner_fn(), loop_factory=get_loop_factory())
  File "/Users/jtomlinson/miniconda3/envs/dask/lib/python3.13/asyncio/runners.py", line 195, in run
    return runner.run(main)
           ~~~~~~~~~~^^^^^^
  File "/Users/jtomlinson/miniconda3/envs/dask/lib/python3.13/asyncio/runners.py", line 118, in run
    return self._loop.run_until_complete(task)
           ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^
  File "/Users/jtomlinson/miniconda3/envs/dask/lib/python3.13/asyncio/base_events.py", line 725, in run_until_complete
    return future.result()
           ~~~~~~~~~~~~~^^
  File "/Users/jtomlinson/Projects/dask/distributed/distributed/utils_test.py", line 377, in inner_fn
    return await async_fn(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/jtomlinson/Projects/dask/distributed/distributed/utils_test.py", line 1089, in async_fn_outer
    return await utils_wait_for(async_fn(), timeout=timeout * 2)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/jtomlinson/Projects/dask/distributed/distributed/utils.py", line 1930, in wait_for
    return await fut
           ^^^^^^^^^
  File "/Users/jtomlinson/Projects/dask/distributed/distributed/utils_test.py", line 1021, in async_fn
    result = await coro2
             ^^^^^^^^^^^
  File "/Users/jtomlinson/Projects/dask/distributed/distributed/utils.py", line 1930, in wait_for
    return await fut
           ^^^^^^^^^
  File "/Users/jtomlinson/Projects/dask/distributed/distributed/tests/test_scheduler.py", line 1161, in test_restart_not_all_workers_return
    with pytest.raises(TimeoutError, match="Waited for 2 worker"):
         ~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/jtomlinson/miniconda3/envs/dask/lib/python3.13/site-packages/_pytest/raises.py", line 721, in __exit__
    raise AssertionError(self._fail_reason)
AssertionError: Regex pattern did not match.
 Regex: 'Waited for 2 worker'
 Input: '[Errno None] None'
============================================================ slowest 20 durations ============================================================
1.21s call     distributed/tests/test_scheduler.py::test_restart_not_all_workers_return

(2 durations < 0.005s hidden.  Use -vv to show these durations.)
========================================================== short test summary info ===========================================================
FAILED distributed/tests/test_scheduler.py::test_restart_not_all_workers_return - AssertionError: Regex pattern did not match.
============================================================= 1 failed in 2.11s ==============================================================

  6. jacobtomlinson commented on Nov 11, 2025

    @jacobtomlinson
    jacobtomlinson
    on Nov 11, 2025
    MemberAuthor

    I can reproduce this as far back as dask/distributed 2023.1.0 (I could probably keep going but would need to downgrade things like numpy). So something else in the environment must've changed to cause this.

  7. jacobtomlinson commented on Nov 11, 2025

    @jacobtomlinson
    jacobtomlinson
    on Nov 11, 2025
    MemberAuthor

    Downgrading tblib to 3.1.0 resolves the issue.

    These two runs are testing the same distributed code, but one passes and the other fails. There are a few environment differences but downgrading tblib was the one that got things passing locally for me.

    Passed on main - https://github.com/dask/distributed/actions/runs/19143357901/job/54714340720
    Failed in CI - https://github.com/dask/distributed/actions/runs/19269845290/job/55094960448?pr=9141

  8. jacobtomlinson commented on Nov 12, 2025

    @jacobtomlinson
    jacobtomlinson
    on Nov 12, 2025
    MemberAuthor

    I've opened ionelmc/python-tblib#85 upstream to hopefully resolve this in tblib. In the meantime we could exclude 3.2.0 and 3.2.1 to get CI passing again.

  9. jacobtomlinson commented on Nov 12, 2025

    @jacobtomlinson
    jacobtomlinson
    on Nov 12, 2025
    MemberAuthor

    ionelmc/python-tblib#85 has been merged and tblib==3.2.2 was released. I verified that the failing tests are passing again locally. Regardless of whether we choose to merge #9141 we can close this now.

  10. jacobtomlinson
    closed this as completedon Nov 12, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    flaky testIntermittent failures on CI.

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions