Repository navigation
fix(levelcode): stop billing streams that failed upstream, and keep a silent stream alive - #452
Merged
Merged
Conversation
Three cases, run against the unchanged gateway: a provider error before any content releases the reservation, a provider error after the usage chunk bills the real usage, and a client that hangs up after content is charged the reserved estimate. Spec-only, so the next commit's behaviour change shows up as a diff to these.
…ailed upstream finalize_stream! treated 'content streamed, no usage chunk' as a client hang-up and charged the reservation, which is the worst case for the turn (input plus the full output ceiling). A provider that errors after streaming some content, or whose connection drops (EOFError is an IOError, so it landed in the hang-up rescue), left the customer paying that estimate for a turn OpenRouter itself billed at $0.00. Seen on Gemini 4 Argon / Google Vertex. Only a failed write to the client is a hang-up now: tee_chunk remembers it. A failure on the provider's or our side releases the reservation. A usage chunk that did arrive is still billed at its real cost, and a client that walks away after content is still charged, both pinned in the previous commit.
…silent A reasoning model can think for minutes before its first token, and the gateway sent the editor nothing meanwhile. The proxy in front of the app closes an idle connection (nginx's proxy_read_timeout defaults to 60 s), which the editor shows as a bare 'terminated' with no explanation. Gemini 4 Argon on Google Vertex streamed at about 1 token a second on 2026-10-15, so a single turn ran past two minutes. While the stream is quiet for KEEPALIVE_INTERVAL (15 s) the gateway writes an SSE comment. The editor only reads 'data:' lines, so it ignores it; the proxy sees traffic. The thread is joined before the stream closes, so nothing lands after [DONE], and an upstream error frame still arrives after the silence.
Contributor
There was a problem hiding this comment.
🟡 Changes recommended
Keep-alive lifecycle handling permits writes after [DONE] and skips reservation cleanup if thread startup fails.
2 open findings
What changed in this PR
Updates streaming chat to release reservations on upstream failures and keep silent provider connections alive.
Changes:
- Distinguishes provider failures from client disconnects for billing.
- Adds periodic SSE keep-alive comments.
- Adds settlement and keep-alive regression tests.
| File | Description |
|---|---|
| spec/requests/api/levelcode/v1/ai_spec.rb | Tests billing outcomes and keep-alive behavior. |
| app/controllers/api/levelcode/v1/ai_controller.rb | Updates failure settlement and adds keep-alive lifecycle handling. |
🧠 Review effort: Balanced
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
start_keepalive ran after the reservation was taken but before the begin/ensure that releases it, so anything it raised (ThreadError under thread exhaustion) skipped settlement and never closed the stream. It now runs inside the block. It also no longer costs the customer the turn when it cannot start: thread creation sits behind spawn_keepalive_thread, and a failure there is logged and the turn streams without a keep-alive. Review finding on #452.
stop_keepalive ran only in the ensure, after [DONE] had been written. A worker that had already passed its idle check could still write its comment after the terminator, and joining it afterwards only waited for that late write. The new spec holds the worker mid-write as the stream ends and saw 'data: [DONE]' followed by ': keepalive'. stop_keepalive is now called immediately before [DONE] and is idempotent, so the ensure still calls it on every error path. Review finding on #452.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Two fixes to
Api::Levelcode::V1::AiController#stream_chat, from what Gemini 4 Argon did on Google Vertex on 2026-10-15. They are separate commits so either can be reverted alone.What was seen
OpenRouter's log for a failing Argon turn: one upstream request,
Attempts 1 (0 failed), a 200 after 2.2 s, then 187 tokens over 126 s (1.5 tok/s), recorded with a 504 and a cost of $0.00. The editor showed a bareterminated. The turn's footer still showed a credit charge.1. A stream that failed upstream no longer bills the reservation (69a3d6c)
finalize_stream!treated "content streamed, no usage chunk" as the client having hung up and charged the reserved estimate, which is the worst case for the turn (input plus the full output ceiling). Two ways that fired for a turn the customer did not get:UpstreamError);EOFErroris anIOError, so it landed in the same rescue as a client hang-up.Reproduced in a spec before the fix: the ledger got a 5,000-micro charge for a turn OpenRouter billed at $0.
Now only a failed write to the client is a hang-up (
tee_chunkremembers it). A failure on the provider's side or ours releases the reservation.The first three rows are pinned in 197ad60 and pass on the unchanged code.
2. Keep-alive while the provider is silent (3ac3911)
The adapter drops every line that is not
data:, so while a reasoning model thinks the editor gets nothing. The proxy in front of the app closes an idle connection (nginx'sproxy_read_timeoutis 60 s by default), which the editor reports asterminated.While the stream has been quiet for
KEEPALIVE_INTERVAL(15 s) the gateway writes: keepalive. The editor reads onlydata:lines (openaiCompat.jsreturns early on anything else), so it ignores the comment. The thread is joined before the stream closes, so nothing lands after[DONE], and an upstream error frame still arrives after a silence.Not changed
READ_TIMEOUTstays 300 s. It is a silence limit already, and shortening it would cut reasoning models that legitimately think for minutes.terminatedfor a stream that ends without[DONE]or an error frame.Tests
spec/requests/api/levelcode/v1/ai_spec.rb: 45 to 54 examples, 0 failures.spec/services/levelcode,levelcode_plans_spec,levelcode_model_catalog_spec: 478 examples, 0 failures.&& !@stream_failed(2 failures), not remembering a failed client write (1), treating everyIOErroras a hang-up again (1), never stopping the keep-alive thread (1), never starting it (2).Not verified
terminated.