Repository navigation
fix(harness): recover from provider errors, broken streams and context overflow - #268
Merged
Merged
Conversation
Part of #262. Each item below came with a test that failed before the change. Cache and request shape: - The proxy adds and signs the Claude Code billing line only for models whose name starts with `claude` (reusing attributionHeaderPolicy). The line changed on every request, so OpenAI and GLM could only reuse the cached tool definitions. Measured on OpenRouter, later turns now hit the cache for about 99% of the input. - Responses requests carry the CLI session id as `prompt_cache_key`; requests with tools send `parallel_tool_calls: true`; tool schemas drop the `$schema` URL and zod's safe-integer bounds. - Cache hits are also read from `prompt_cache_hit_tokens` and a top-level `cached_tokens`. Responses report the requested model name. Transport: - The proxy route turns off Bun's 60-second idle timeout per request. - A base URL that already ends in a version (`/v4`, `/v1beta/openai`) is used as the full prefix for requests, model lists and connection tests; others still get `/v1`. - The caller's abort signal reaches the upstream fetch, also before the upstream answers and for non-streaming requests. - With a manual HTTP proxy, the CLI env gets NO_PROXY for loopback, merged with the inherited value (an inherited `*` is kept as is). Scheduled tasks now get the network env too. - One stream supervisor times idle gaps (aiRequestTimeoutMs; only events sent to the CLI count, keep-alive comments become `ping`) and records how the stream ended. The CLI direct path gets the same limit through CLAUDE_ENABLE_STREAM_WATCHDOG and CLAUDE_STREAM_IDLE_TIMEOUT_MS. - Every upstream request is recorded as an `upstream_request` diagnostics event (provider, model, host, status, timings, request id, outcome); never the key or the body. diagnosticsService resolves its paths before awaiting, so a pending write cannot land in another config dir. - All three SSE readers share one spec framing function; a formatted JSON error body is surfaced instead of an empty reply. - OpenAI-format providers can carry extraHeaders and queryParams; configured headers go first, auth and Content-Type override them. They are rejected for anthropic-format providers. No desktop form yet. - The proxy answers count_tokens and other unknown paths with an Anthropic not_found_error. The server marks proxied providers with DREAMCODER_PROVIDER_PROXIED, and /context uses the local estimate for them instead of a Haiku request. Docs: CHANGELOG entries, status lines on the fixed items in docs/features/harness/proxy, measured cache results in the harness README.
…t overflow - Retry a stream that breaks mid-reply by opening a new stream, sharing one budget with request retries; no non-streaming fallback. The partial reply is retracted from history, the transcript and connected clients. - The proxy parses provider errors into Anthropic error types, passes retry-after and request ids through, and marks quota and auth failures as not retryable; the CLI classifies them with the same table. - Context overflow from OpenAI-format providers is recognised as prompt-too-long so reactive compaction runs. - Truncated tool arguments are refused with an explanation to the model instead of running the tool with an empty input. Part of #263
This was referenced Nov 18, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Second batch of the proxy-layer fixes from #261: what happens when a session grows or a provider call goes wrong.
What changed
Broken streams are retried as streams (P-RES-04, P-RES-06, P-RES-12)
CLAUDE_CODE_DISABLE_NONSTREAMING_FALLBACK=1for every provider, so a stream error no longer falls back to a non-streaming request capped at 120 s.queryModelinsrc/services/api/claude.tsreopens the stream when it breaks mid-reply. Stream retries and request retries share one budget ofgetDefaultMaxRetries() + 1requests. Errors the provider will repeat (invalid request, auth, quota) and user aborts are not retried.errorwhen a chat stream ends withoutfinish_reasonor[DONE], a Responses stream ends without a terminal event, or a reply has no content.The partial reply is retracted (three places)
QueryEnginedrops the partial assistant messages from history and from the transcript, and emits astream_retractedsystem message beforeapi_retry.translate.tsturns it into a newreply_retractedWebSocket message and sets the turn activity back tothinkingthrough the reducer.chatStoreremoves what the failed attempt streamed in the current reply.docs/features/session-state/tech-design.mdare updated.Provider errors keep their meaning (P-NET-02, P-NET-03, P-NET-04, P-RES-05)
src/utils/providerErrors.tsis one table of overflow, quota, auth and rate-limit codes and messages, used by both the proxy and the CLI.retry-afterand the request id through, maps quota errors to 402billing_error, and setsx-should-retry: falseon errors that must not be retried. Error bodies inside a 2xx reply are handled the same way.content_filter,sensitive,network_errorandmodel_context_window_exceeded, and Responsesincomplete/failed, map to errors or the right stop reason instead ofend_turn/max_tokens.HTTP <status>: <message>.Usage (P-RES-01): missing usage is no longer written as 0, and Kimi's
choices[0].usageis read.Stream conversion (P-RES-08, P-RES-09): one chat delta carrying text, reasoning and tool calls is handled in full; the Responses converter reads completed items and closes blocks on completion.
Truncated tool arguments (P-RES-07): arguments that are not a complete JSON object are not run. Both paths mark them with one reserved key, and the model gets an error result saying they were cut off. Blank and
nullarguments still mean no arguments.Not done here: on the direct (Anthropic-format) path, a stream that closes after its blocks but before
message_deltastill counts as a normal end; recorded as a follow-up indocs/features/harness/proxy/response.mdP-RES-04. Timeouts are unchanged and documented as a known limitation. Tools still run after the reply has finished streaming (tengu_streaming_tool_execution2is always off in DreamCoder), so a retraction never has to undo a tool that already ran; noted indocs/features/harness/comparison/loop-execution.md.How it was tested
Each item was written test-first; the new tests fail on
main.streamRetry.test.ts(retry, no retry on invalid request, shared budget, user abort),queryEngineRetraction.test.ts,stream-retraction.test.ts,providerRetry.test.ts,contextOverflow.test.ts,truncatedToolArguments.test.ts, plus cases in the proxy, upstream-client, providers and desktopchatStoretests.bun test --randomize --seed=1: 405 pass.bun testlocally: 2900 pass, 11 fail; the same 11 fail onmainon this machine (environment: ripgrep, local Claude Code config).bun run lintpasses,bun run test -- --run3119 pass.anthropicStream.tswas checked with the upstream-client and proxy test files (174 pass).Closes #263