Repository navigation
Closed
Labels
bugSomething isn't workingSomething isn't workingenhancementNew feature or requestNew feature or request
Description
Part of #261. First of three batches for the proxy layer between the CLI and third-party providers (src/server/proxy, src/server/upstream, provider settings).
This batch fixes the request path itself: things that are confirmed in code or by local experiments, are independent of each other, and do not need per-model knowledge. Each item links to its finding, which has the current behavior, a concrete scenario, how Codex CLI and ZCode handle it, and a fix direction.
Before and after
Measure on the same task set before and after this batch: cache hit tokens, input tokens and turns per task. The cache numbers need P-RES-11 to be read correctly, so land that first or with the billing-line fix.
High
- Billing line breaks prefix caching (P-REQ-01). The proxy prepends an
x-anthropic-billing-headerline whosecchis a hash of the whole request body, so the start of every request differs. The server already disables this line for non-Claude models (CLAUDE_CODE_ATTRIBUTION_HEADER=0) but the proxy ignores that setting. Nothing in the repository reads the line. - Requests are cut after 60 seconds without output (P-NET-01).
Bun.serveruns withidleTimeout: 60(src/server/index.ts:155). A reasoning model thinking silently, a slow first byte or a non-streaming request is cut off, then retried up to 10 times and billed each time. The "AI request timeout" setting has no effect above 60 seconds on this path. - No idle timeout once a stream has started (P-RES-03, P-NET-08). A stalled upstream stream hangs the turn forever; the direct Anthropic-format path has the same gap. Use one explicit upstream idle timeout for both paths and turn upstream keep-alive comments into
pingevents.
Medium
- Base URL is forced to
/v1(P-REQ-06, P-NET-06).normalizeUpstreamBaseUrlstrips/v1and the client appends/v1/..., so Zhipupaas/v4, Volcengine Arkv3and Gemini's OpenAI-compatible endpoint return 404, including "Test connection". Saved configs with an empty path keep working. - Stopping a turn does not cancel the upstream request (P-NET-05, P-RES-13). Non-streaming requests and requests still waiting for headers run to completion and are billed. Pass
req.signal(combined with the timeout signal) to the upstream fetch. - Manual proxy mode sends local proxy traffic to the external proxy (P-NET-07).
NO_PROXYis not set, so the CLI's requests to127.0.0.1go through the user's HTTP proxy and fail for remote corporate proxies. - No cache key or session header upstream (P-REQ-10). The CLI already sends
X-Claude-Code-Session-Id; map it toprompt_cache_keyor a session header where the provider supports it. Only useful once the billing line is fixed.
Low
- Cached tokens are read from one field only (P-RES-11). Also read DeepSeek's
prompt_cache_hit_tokensand Moonshot'scached_tokens. - Providers cannot set extra headers or query parameters (P-NET-11), for example Azure API versions or OpenRouter attribution headers.
- Proxy logs cannot diagnose a failed request (P-NET-12). Record provider, model, duration, status and upstream request id per request, without keys; mid-stream errors and disconnects are not logged today.
-
count_tokensis not supported (P-NET-13). In the terminal,/contextfalls back to onemax_tokens: 1request per item, and the proxy dropsmax_tokens, so each becomes a full generation. Answercount_tokenslocally with an estimate. - SSE parsing is not spec-complete (P-RES-14). One shared framing function for both converters:
data:without a space, multi-linedata:, non-SSE error bodies on the Responses path. - Model name in responses comes from three different sources (P-RES-15). Use the requested model name everywhere.
-
parallel_tool_callsis never sent (P-REQ-12). Some providers may default it to false (unverified). - Tool schemas are forwarded as is (P-REQ-13), including
$schemaandmaximum: 9007199254740991. Strip keywords providers do not use.
Overview of all three batches: docs/features/harness.
Metadata
Metadata
Assignees
Labels
bugSomething isn't workingSomething isn't workingenhancementNew feature or requestNew feature or request
Activity
fix(proxy): fix the basics of the OpenAI-format request path
fix(proxy): fix the basics of the OpenAI-format request path (#266)
GoDiao commentedon Nov 18, 2026
Merged in #266 (ce209f6). 14 of the 15 items are fixed, each with a test.
Two parts are not done and move to #267:
extraHeadersandqueryParams. The settings form does not offer them yet, so they can only be set by editingproviders.json.sendToUpstreamon OpenRouter: gpt-4.1-mini hits about 99% of the input from turn 2, glm-4.6 hits 13312 of 13371 input tokens on turn 3. The task-set comparison of cache hits, input tokens and turns per task is still to do.fix(harness): recover from provider errors, broken streams and contex…