Steps to reproduce
- Deploy a long-context model as a service on a single GPU, e.g.
Qwen/Qwen3.8-27B-FP8 on one H100 with --max-model-len 262144 --max-num-seqs 16, behind a dstack gateway.
- Send ~30 concurrent streaming chat-completion requests, each with 60k-100k input tokens.
- Wait 5 minutes.
Actual behaviour
At 300s the gateway closes connections mid-body. The client sees a truncated chunked response rather than an HTTP status code:
ClientPayloadError("Response payload is not completed:
<TransferEncodingError: 400, message='Not enough data to satisfy transfer length header.'>")
Timing matched exactly: requests dispatched at 15:41:52, connection cut at 15:46:59.
Nothing was failing upstream. For the whole preceding 300s the load generator reported errors=0, and 22 of the 30 requests had already returned successfully. The remaining requests were simply queued behind ~1.9M tokens of prefill.
The mechanism is that proxy_read_timeout bounds the gap between successive reads from upstream, not total request duration. For a streaming LLM response the engine sends response headers immediately and the first data: chunk only after prefill completes, so this timeout effectively caps time-to-first-token. Under load with 100k-token prompts, TTFT can exceed 300s legitimately. For example, I hit it during benchmark warmup phase that fired ~30 concurrent 100k-token requests at once. Steady-state serving rarely leaves a request queued for five minutes, so this is not a common availability problem, but it may still happen.
Expected behaviour
proxy_read_timeout should be configurable per service, the way client_max_body_size already is via DEFAULT_SERVICE_CLIENT_MAX_BODY_SIZE and a per-service field.
A higher default for services that declare model: would also be reasonable, since long-context inference is exactly the workload where a 300s inter-read gap is reachable without anything being wrong.
dstack version
0.21.3 (master, e828c9e14)
Additional information
Steps to reproduce
Qwen/Qwen3.8-27B-FP8on one H100 with--max-model-len 262144 --max-num-seqs 16, behind adstackgateway.Actual behaviour
At 300s the gateway closes connections mid-body. The client sees a truncated chunked response rather than an HTTP status code:
Timing matched exactly: requests dispatched at 15:41:52, connection cut at 15:46:59.
Nothing was failing upstream. For the whole preceding 300s the load generator reported
errors=0, and 22 of the 30 requests had already returned successfully. The remaining requests were simply queued behind ~1.9M tokens of prefill.The mechanism is that
proxy_read_timeoutbounds the gap between successive reads from upstream, not total request duration. For a streaming LLM response the engine sends response headers immediately and the firstdata:chunk only after prefill completes, so this timeout effectively caps time-to-first-token. Under load with 100k-token prompts, TTFT can exceed 300s legitimately. For example, I hit it during benchmark warmup phase that fired ~30 concurrent 100k-token requests at once. Steady-state serving rarely leaves a request queued for five minutes, so this is not a common availability problem, but it may still happen.Expected behaviour
proxy_read_timeoutshould be configurable per service, the wayclient_max_body_sizealready is viaDEFAULT_SERVICE_CLIENT_MAX_BODY_SIZEand a per-service field.A higher default for services that declare
model:would also be reasonable, since long-context inference is exactly the workload where a 300s inter-read gap is reachable without anything being wrong.dstack version
0.21.3 (master,
e828c9e14)Additional information