Skip to content

Harness: proxy request path fixes #262

Closed

Description

@GoDiao

Part of #261. First of three batches for the proxy layer between the CLI and third-party providers (src/server/proxy, src/server/upstream, provider settings).

This batch fixes the request path itself: things that are confirmed in code or by local experiments, are independent of each other, and do not need per-model knowledge. Each item links to its finding, which has the current behavior, a concrete scenario, how Codex CLI and ZCode handle it, and a fix direction.

Before and after

Measure on the same task set before and after this batch: cache hit tokens, input tokens and turns per task. The cache numbers need P-RES-11 to be read correctly, so land that first or with the billing-line fix.

High

  • Billing line breaks prefix caching (P-REQ-01). The proxy prepends an x-anthropic-billing-header line whose cch is a hash of the whole request body, so the start of every request differs. The server already disables this line for non-Claude models (CLAUDE_CODE_ATTRIBUTION_HEADER=0) but the proxy ignores that setting. Nothing in the repository reads the line.
  • Requests are cut after 60 seconds without output (P-NET-01). Bun.serve runs with idleTimeout: 60 (src/server/index.ts:155). A reasoning model thinking silently, a slow first byte or a non-streaming request is cut off, then retried up to 10 times and billed each time. The "AI request timeout" setting has no effect above 60 seconds on this path.
  • No idle timeout once a stream has started (P-RES-03, P-NET-08). A stalled upstream stream hangs the turn forever; the direct Anthropic-format path has the same gap. Use one explicit upstream idle timeout for both paths and turn upstream keep-alive comments into ping events.

Medium

  • Base URL is forced to /v1 (P-REQ-06, P-NET-06). normalizeUpstreamBaseUrl strips /v1 and the client appends /v1/..., so Zhipu paas/v4, Volcengine Ark v3 and Gemini's OpenAI-compatible endpoint return 404, including "Test connection". Saved configs with an empty path keep working.
  • Stopping a turn does not cancel the upstream request (P-NET-05, P-RES-13). Non-streaming requests and requests still waiting for headers run to completion and are billed. Pass req.signal (combined with the timeout signal) to the upstream fetch.
  • Manual proxy mode sends local proxy traffic to the external proxy (P-NET-07). NO_PROXY is not set, so the CLI's requests to 127.0.0.1 go through the user's HTTP proxy and fail for remote corporate proxies.
  • No cache key or session header upstream (P-REQ-10). The CLI already sends X-Claude-Code-Session-Id; map it to prompt_cache_key or a session header where the provider supports it. Only useful once the billing line is fixed.

Low

  • Cached tokens are read from one field only (P-RES-11). Also read DeepSeek's prompt_cache_hit_tokens and Moonshot's cached_tokens.
  • Providers cannot set extra headers or query parameters (P-NET-11), for example Azure API versions or OpenRouter attribution headers.
  • Proxy logs cannot diagnose a failed request (P-NET-12). Record provider, model, duration, status and upstream request id per request, without keys; mid-stream errors and disconnects are not logged today.
  • count_tokens is not supported (P-NET-13). In the terminal, /context falls back to one max_tokens: 1 request per item, and the proxy drops max_tokens, so each becomes a full generation. Answer count_tokens locally with an estimate.
  • SSE parsing is not spec-complete (P-RES-14). One shared framing function for both converters: data: without a space, multi-line data:, non-SSE error bodies on the Responses path.
  • Model name in responses comes from three different sources (P-RES-15). Use the requested model name everywhere.
  • parallel_tool_calls is never sent (P-REQ-12). Some providers may default it to false (unverified).
  • Tool schemas are forwarded as is (P-REQ-13), including $schema and maximum: 9007199254740991. Strip keywords providers do not use.

Overview of all three batches: docs/features/harness.

Activity

  1. GoDiao commented on Nov 18, 2026

    @GoDiao
    OwnerAuthor

    Merged in #266 (ce209f6). 14 of the 15 items are fixed, each with a test.

    Two parts are not done and move to #267:

    • Extra headers and query parameters (P-NET-11): the server side is done; OpenAI-format providers can carry extraHeaders and queryParams. The settings form does not offer them yet, so they can only be set by editing providers.json.
    • Before and after: only cache hits were measured, on multi-turn requests through sendToUpstream on OpenRouter: gpt-4.1-mini hits about 99% of the input from turn 2, glm-4.6 hits 13312 of 13371 input tokens on turn 3. The task-set comparison of cache hits, input tokens and turns per task is still to do.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't workingenhancementNew feature or request

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions