fix: resolve result_future on inference error instead of tearing down the model - #157
Open
AmirF194 wants to merge 1 commit into
Open
Conversation
… the model QueueWorker.queue_worker_llm (and the identical loop in every other worker: vlm, whisper, qwen3_asr, kokoro, qwen3_tts, emb, rr) broke out of its processing loop on an inference error before ever calling packet.result_future.set_result(...). Non-streaming callers such as WorkerRegistry.generate() await that future, so a failed generation hung the HTTP caller forever. The same branch also fired an unconditional registry.register_unload(model_name), whose destructor can touch a corrupted device context after certain errors and SIGABRT the whole process, taking every other loaded model down with it. Each worker now resolves result_future with the error via set_exception(RuntimeError(...)) and continues its loop instead of breaking, so a failed request surfaces as an error to its caller and the model/worker stay alive for the next request. The EMB and RR workers use a different failure condition (an empty response, since InferWorker.infer_emb tags a raised exception as a truthy "Error: ..." string) but had the exact same break-before-resolve bug. Fixes SearchSavior#153
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Root cause
QueueWorker.queue_worker_llmand the identical loop repeated in every otherworker (
vlm,whisper,qwen3_asr,kokoro,qwen3_tts,emb,rr)broke out of its
while Trueloop on an inference error before ever callingpacket.result_future.set_result(...). Non-streaming callers such asWorkerRegistry.generate()awaitthat future, so a failed generation hungthe caller forever. The same branch also fired an unconditional
registry.register_unload(model_name); that unload's pipeline destructor cantouch a corrupted device context after certain errors and abort the whole
process, taking every other loaded model down with it. This matches the issue
exactly for the LLM worker; I confirmed by reading each of the other seven
worker functions that they share the identical structural bug (
grep/astenumeration in
src/server/worker_registry.py), not only the ones named inthe issue as "likely" affected.
embandrrtrigger on a different condition (not completed_packet.response,since
InferWorker.infer_emb/infer_rerankalready tag a raised exception asa truthy
"Error: ..."string, so that branch is actually reached only whenthe pipeline returns no data with no exception), but the same break-before-resolve
bug is there too.
Fix
Each worker now reports the failure through
result_future.set_exception(...)and
continues its loop instead ofbreaking, so a failed request surfaces asan error to its caller and the worker/model stay alive to serve the next
request, per the issue's suggested design.
Verification
tests/unit/test_worker_registry_error_handling.pydrives the real(unmocked)
queue_worker_*loop for all 8 workers with a failing requestfollowed by a healthy one. Confirmed fails on unmodified
main(eachtest times out waiting on
result_future, matching the reported hang) andpasses on this branch, in a clean
python:3.12-slimcontainer.uv run pytest -W ignore tests/unit: 119 passed (was 111), noregressions.
coverage run --source=src.server.worker_registry: every line this diffchanges is exercised by the new tests.
SIGABRTitself, since that needs a real IntelGPU/OpenCL context corrupted by a genuine device error, which this session
doesn't have. This PR removes the
register_unloadcall on the error pathentirely (the mechanism the issue names as the crash trigger), but I can't
reproduce the abort to confirm the crash is gone, only that the code no
longer calls the destructor on that path.
Fixes #153