Not sure what's causing this, but I've encountered this a handful of times so far.
This exception shows up on the distributed-to-runners log. Meanwhile, the orchestrator is basically just waiting forever for something that will never come.
After this exception shows up on the runner, I can't cancel the job in the orchestrators queue, it remains hung and "Running" forever. Issuing cancel says [INFO] Cancelling running prompt <uuid>, but doesn't actually cancel it. Needs me to restart the master fully.
[Distributed] Worker - Failed to send canonical image/audio envelope to master: 404, message='Not Found', url='http://{ip}:{port}/distributed/job_complete'
[ERROR] !!! Exception during processing !!! 404, message='Not Found', url='http://{ip}:{port}/distributed/job_complete'
[ERROR] Traceback (most recent call last):
File "{root}/ComfyUI/execution.py", line 545, in execute
output_data, output_ui, has_subgraph, has_pending_tasks = await get_output_data(prompt_id, unique_id, obj, input_data_all, execution_block_cb=execution_block_cb, pre_execute_cb=pre_execute_cb, v3_data=v3_data)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "{root}/ComfyUI/execution.py", line 344, in get_output_data
return_values = await _async_map_node_over_list(prompt_id, unique_id, obj, input_data_all, obj.FUNCTION, allow_interrupt=True, execution_block_cb=execution_block_cb, pre_execute_cb=pre_execute_cb, v3_data=v3_data)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "{root}/ComfyUI/execution.py", line 312, in _async_map_node_over_list
await process_inputs(input_data_all, 0, input_is_list=input_is_list)
File "{root}/ComfyUI/execution.py", line 306, in process_inputs
result = f(**inputs)
^^^^^^^^^^^
File "{root}/ComfyUI/custom_nodes/ComfyUI-Distributed/nodes/collector.py", line 138, in run
result = run_async_in_server_loop(
^^^^^^^^^^^^^^^^^^^^^^^^^
File "{root}/ComfyUI/custom_nodes/ComfyUI-Distributed/utils/async_helpers.py", line 53, in run_async_in_server_loop
raise error
File "{root}/ComfyUI/custom_nodes/ComfyUI-Distributed/utils/async_helpers.py", line 38, in wrapper
result = await coro
^^^^^^^^^^
File "{root}/ComfyUI/custom_nodes/ComfyUI-Distributed/nodes/collector.py", line 327, in execute
await self.send_batch_to_master(images, audio, multi_job_id, master_url, worker_id)
File "{root}/ComfyUI/custom_nodes/ComfyUI-Distributed/nodes/collector.py", line 199, in send_batch_to_master
response.raise_for_status()
File "{root}/lib/python3.12/site-packages/aiohttp/client_reqrep.py", line 636, in raise_for_status
raise ClientResponseError(
aiohttp.client_exceptions.ClientResponseError: 404, message='Not Found', url='http://{ip}:{port}/distributed/job_complete'
Not sure what's causing this, but I've encountered this a handful of times so far.
This exception shows up on the distributed-to-runners log. Meanwhile, the orchestrator is basically just waiting forever for something that will never come.
After this exception shows up on the runner, I can't cancel the job in the orchestrators queue, it remains hung and "Running" forever. Issuing cancel says
[INFO] Cancelling running prompt <uuid>, but doesn't actually cancel it. Needs me to restart the master fully.