Skip to content

fix(harness): recover from provider errors, broken streams and context overflow - #268

Merged
GoDiao merged 6 commits into
mainfrom
feat/harness-finetune
Nov 18, 2026
Merged

GoDiao merged 6 commits into
mainfrom
feat/harness-finetune

Conversation

@GoDiao

@GoDiao GoDiao commented Nov 18, 2026

Copy link
Copy Markdown
Owner

Second batch of the proxy-layer fixes from #261: what happens when a session grows or a provider call goes wrong.

What changed

Broken streams are retried as streams (P-RES-04, P-RES-06, P-RES-12)

  • The server sets CLAUDE_CODE_DISABLE_NONSTREAMING_FALLBACK=1 for every provider, so a stream error no longer falls back to a non-streaming request capped at 120 s.
  • queryModel in src/services/api/claude.ts reopens the stream when it breaks mid-reply. Stream retries and request retries share one budget of getDefaultMaxRetries() + 1 requests. Errors the provider will repeat (invalid request, auth, quota) and user aborts are not retried.
  • The proxy emits an SSE error when a chat stream ends without finish_reason or [DONE], a Responses stream ends without a terminal event, or a reply has no content.

The partial reply is retracted (three places)

  • CLI: QueryEngine drops the partial assistant messages from history and from the transcript, and emits a stream_retracted system message before api_retry.
  • Server: translate.ts turns it into a new reply_retracted WebSocket message and sets the turn activity back to thinking through the reducer.
  • Desktop and H5: chatStore removes what the failed attempt streamed in the current reply.
  • Protocol types and docs/features/session-state/tech-design.md are updated.

Provider errors keep their meaning (P-NET-02, P-NET-03, P-NET-04, P-RES-05)

  • src/utils/providerErrors.ts is one table of overflow, quota, auth and rate-limit codes and messages, used by both the proxy and the CLI.
  • The proxy parses the upstream body into an Anthropic error type, passes retry-after and the request id through, maps quota errors to 402 billing_error, and sets x-should-retry: false on errors that must not be retried. Error bodies inside a 2xx reply are handled the same way.
  • Context overflow from OpenAI, DashScope, Moonshot, Zhipu and others becomes Anthropic's prompt-too-long, so reactive compaction runs.
  • The CLI no longer retries quota errors off the first-party API, and retries 401 only when credentials can be refreshed.
  • Finish reasons such as content_filter, sensitive, network_error and model_context_window_exceeded, and Responses incomplete/failed, map to errors or the right stop reason instead of end_turn/max_tokens.
  • Provider connection test, model discovery and title generation show HTTP <status>: <message>.

Usage (P-RES-01): missing usage is no longer written as 0, and Kimi's choices[0].usage is read.

Stream conversion (P-RES-08, P-RES-09): one chat delta carrying text, reasoning and tool calls is handled in full; the Responses converter reads completed items and closes blocks on completion.

Truncated tool arguments (P-RES-07): arguments that are not a complete JSON object are not run. Both paths mark them with one reserved key, and the model gets an error result saying they were cut off. Blank and null arguments still mean no arguments.

Not done here: on the direct (Anthropic-format) path, a stream that closes after its blocks but before message_delta still counts as a normal end; recorded as a follow-up in docs/features/harness/proxy/response.md P-RES-04. Timeouts are unchanged and documented as a known limitation. Tools still run after the reply has finished streaming (tengu_streaming_tool_execution2 is always off in DreamCoder), so a retraction never has to undo a tool that already ran; noted in docs/features/harness/comparison/loop-execution.md.

How it was tested

Each item was written test-first; the new tests fail on main.

  • New: streamRetry.test.ts (retry, no retry on invalid request, shared budget, user abort), queryEngineRetraction.test.ts, stream-retraction.test.ts, providerRetry.test.ts, contextOverflow.test.ts, truncatedToolArguments.test.ts, plus cases in the proxy, upstream-client, providers and desktop chatStore tests.
  • Touched test files with bun test --randomize --seed=1: 405 pass.
  • Full bun test locally: 2900 pass, 11 fail; the same 11 fail on main on this machine (environment: ripgrep, local Claude Code config).
  • Desktop: bun run lint passes, bun run test -- --run 3119 pass.
  • The last review round's change to anthropicStream.ts was checked with the upstream-client and proxy test files (174 pass).

Closes #263

Part of #262. Each item below came with a test that failed before the
change.

Cache and request shape:
- The proxy adds and signs the Claude Code billing line only for models
  whose name starts with `claude` (reusing attributionHeaderPolicy). The
  line changed on every request, so OpenAI and GLM could only reuse the
  cached tool definitions. Measured on OpenRouter, later turns now hit the
  cache for about 99% of the input.
- Responses requests carry the CLI session id as `prompt_cache_key`;
  requests with tools send `parallel_tool_calls: true`; tool schemas drop
  the `$schema` URL and zod's safe-integer bounds.
- Cache hits are also read from `prompt_cache_hit_tokens` and a top-level
  `cached_tokens`. Responses report the requested model name.

Transport:
- The proxy route turns off Bun's 60-second idle timeout per request.
- A base URL that already ends in a version (`/v4`, `/v1beta/openai`) is
  used as the full prefix for requests, model lists and connection tests;
  others still get `/v1`.
- The caller's abort signal reaches the upstream fetch, also before the
  upstream answers and for non-streaming requests.
- With a manual HTTP proxy, the CLI env gets NO_PROXY for loopback, merged
  with the inherited value (an inherited `*` is kept as is). Scheduled
  tasks now get the network env too.
- One stream supervisor times idle gaps (aiRequestTimeoutMs; only events
  sent to the CLI count, keep-alive comments become `ping`) and records how
  the stream ended. The CLI direct path gets the same limit through
  CLAUDE_ENABLE_STREAM_WATCHDOG and CLAUDE_STREAM_IDLE_TIMEOUT_MS.
- Every upstream request is recorded as an `upstream_request` diagnostics
  event (provider, model, host, status, timings, request id, outcome);
  never the key or the body. diagnosticsService resolves its paths before
  awaiting, so a pending write cannot land in another config dir.
- All three SSE readers share one spec framing function; a formatted JSON
  error body is surfaced instead of an empty reply.
- OpenAI-format providers can carry extraHeaders and queryParams;
  configured headers go first, auth and Content-Type override them. They
  are rejected for anthropic-format providers. No desktop form yet.
- The proxy answers count_tokens and other unknown paths with an Anthropic
  not_found_error. The server marks proxied providers with
  DREAMCODER_PROVIDER_PROXIED, and /context uses the local estimate for
  them instead of a Haiku request.

Docs: CHANGELOG entries, status lines on the fixed items in
docs/features/harness/proxy, measured cache results in the harness README.
…t overflow

- Retry a stream that breaks mid-reply by opening a new stream, sharing one
  budget with request retries; no non-streaming fallback. The partial reply
  is retracted from history, the transcript and connected clients.
- The proxy parses provider errors into Anthropic error types, passes
  retry-after and request ids through, and marks quota and auth failures as
  not retryable; the CLI classifies them with the same table.
- Context overflow from OpenAI-format providers is recognised as
  prompt-too-long so reactive compaction runs.
- Truncated tool arguments are refused with an explanation to the model
  instead of running the tool with an empty input.

Part of #263
@GoDiao
GoDiao merged commit 62501c2 into main Nov 18, 2026
4 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Harness: context overflow, errors and stream robustness in the proxy

1 participant