[Bug]: Gateway buffers streamed LLM responses, inflating TTFT to 16 KB worth of tokens

[r4victor]

Steps to reproduce

1. Deploy a vLLM or SGLang service behind a gateway with https enabled and a model configured.

2. Send a streaming chat completion through the gateway and watch the output arrive:

   curl -N -sS https://<run>.<gateway-domain>/v1/chat/completions \
     -H "Authorization: Bearer $DSTACK_TOKEN" -H "Content-Type: application/json" \
     -d '{"model": "<model>", "stream": true, "max_tokens": 200, "messages": [{"role": "user", "content": "Write a long story"}]}'

3. Run dstack attach <run> and send the same request to the forwarded port on localhost.

Actual behaviour

Through the gateway the response does not stream. Nothing arrives, not even the response headers, until about 16 KB of SSE has accumulated, roughly 65 OpenAI-format chunks. Shorter responses arrive all at once when generation finishes. Directly against the replica, chunks arrive as they are generated.

Measured with a mock OpenAI-compatible service that emits one 247-byte SSE chunk every 250 ms, first chunk after 0.5 s, each chunk stamped with its server-side send time:

• Direct: TTFT 0.5 s, one chunk per read.
• Gateway, 60 tokens: TTFT 16.5 s, the whole response in one read at the end.
• Gateway, 200 tokens: TTFT 17.5 s, then bursts of 64 to 66 chunks every 16.5 s.
• Gateway, same service sending X-Accel-Buffering: no: TTFT 0.5 s plus request overhead, one chunk per read.

Cause: the service location @ block uses the nginx default proxy_buffering on. In that mode nginx releases data only when a 4 KB proxy buffer fills or the response ends, and over HTTPS the 16 KB ssl_buffer_size output buffer sits on top of that, so the effective threshold becomes 16 KB.

dstack/src/dstack/_internal/proxy/gateway/resources/nginx/service.jinja2

Lines 76 to 95 in 7823319

	location @ {
	set $dstack_replica_hit 1;
	{% if replicas %}
	{% if cors_enabled %}
	proxy_hide_header 'Access-Control-Allow-Origin';
	proxy_hide_header 'Access-Control-Allow-Methods';
	proxy_hide_header 'Access-Control-Allow-Headers';
	proxy_hide_header 'Access-Control-Allow-Credentials';
	add_header 'Access-Control-Allow-Origin' '*' always;
	add_header 'Access-Control-Allow-Methods' 'GET, POST, PUT, DELETE, PATCH, OPTIONS, HEAD' always;
	add_header 'Access-Control-Allow-Headers' '*' always;
	{% endif %}
	proxy_pass http://{{ domain }}.upstream;
	proxy_set_header X-Real-IP $remote_addr;
	proxy_set_header Host $host;
	proxy_read_timeout 300s;
	{% else %}
	return 503;
	{% endif %}
	}

vLLM and SGLang do not send X-Accel-Buffering: no on streaming responses, so nothing tells nginx to bypass buffering. TGI does send it, added in huggingface/text-generation-inference#498 for exactly this reason.

https://github.com/vllm-project/vllm/blob/6fbb00b18874e27ba7d7adc0a3b8e93fee763ab1/vllm/entrypoints/openai/chat_completion/api_router.py#L77-L80

The model entrypoint (gateway.<domain>/proxy/models/<project>/) is affected as well. Its first hop is fine because the gateway app sets X-Accel-Buffering: no on streaming responses, but the app forwards requests to the service through the same nginx location @, so the second hop buffers.

dstack/src/dstack/_internal/proxy/lib/routers/model_proxy.py

Lines 61 to 65 in 7823319

	return StreamingResponse(
	await StreamingAdaptor(client.stream(body)).get_stream(),
	media_type="text/event-stream",
	headers={"X-Accel-Buffering": "no"},
	)

Expected behaviour

Streamed tokens reach the client as the replica emits them. TTFT through the gateway should differ from direct access only by the per-request overhead (TLS, auth subrequest, tunnel connection), not by the size of the response.

dstack version

0.21.5

Server logs

No response

Additional information

Proposed fix: disable proxy buffering for services registered with a model, mirroring how cors_enabled is derived in the gateway registry. Add a gateway-only proxy_buffering flag to the Service record, set it from the model at registration, render proxy_buffering off; in location @, and migrate existing services on gateway startup so no redeploy is needed. No server or API changes are required since the gateway already receives the model.

proxy_buffering is per location and cannot depend on the response, so this also disables buffering for the service's non-streaming requests. For those the cost is that a large body to a slow client holds the replica connection for the download instead of being spooled by nginx. Generation is already complete by then, so no engine resources are held. Services without a model keep buffering. A per-service override can be added later if a batch-heavy deployment needs it.

Ideally, we'd like a different buffering behavior depending on the requests's body.stream flag. It's not possible with nginx and is yet another reason to use nginx only as an edge proxy, and have a custom LLM/AI-aware proxy implementation.