Repository navigation
Don't treat capacity phrases in model output as provider failures - #4
Open
rayan-mazouz wants to merge 2 commits into
Open
rayan-mazouz wants to merge 2 commits into
rayan-mazouz wants to merge 2 commits into
Conversation
Capacity phrases ("at capacity", "try a different model", ...) were
matched anywhere in a response, including the model's own text:
- during SSE prefetch the phrase check ran before the output check, so a
healthy stream whose first deltas mentioned one was discarded, the
account was cooled down, and the request was replayed on another
account, spending quota twice;
- mid-stream, a text delta containing one ended the response early;
- a non-streaming message whose text mentioned one triggered failover.
Classify failures per SSE frame instead: a frame is a provider error only
when it carries an error marker and no output event, and it is a capacity
error only when that error frame contains a capacity phrase. Non-streaming
bodies that are Anthropic messages are never errors. Error frames that
arrive before any output still fail over to the next account.
Once a stream had started, an upstream error event was replaced with a synthetic message_stop, so Claude Code received a truncated response (possibly mid tool call) reported as a normal completion and never retried. Buffered non-streaming SSE bodies were rewritten the same way. Failover is only possible before output reaches the client, which the prefetch already handles. After that, pass provider error frames through unchanged so the client sees the failure and can retry. The stream is still re-chunked into whole frames for the model-name rewrite.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Capacity phrases ("at capacity", "try a different model", "service exhausted", "temporarily overloaded") were matched anywhere in a response, including the model's own text:
Separately, a real provider error that arrived after streaming had started was replaced with a synthetic
message_stop. Claude Code therefore got a truncated response, possibly mid tool call, that looked like a normal completion, and it never retried.Commits
messageis never an error. Error frames that arrive before any output still fail over to the next account, exactly as before.Testing
New tests cover:
These tests fail on
main. The full suite passes (155).