Skip to content

fix(proxy): fix the basics of the OpenAI-format request path - #266

Merged
GoDiao merged 4 commits into
mainfrom
feat/harness-finetune
Nov 18, 2026
Merged

GoDiao merged 4 commits into
mainfrom
feat/harness-finetune

Conversation

@GoDiao

@GoDiao GoDiao commented Nov 18, 2026

Copy link
Copy Markdown
Owner

Part of #262. Each item below came with a test that failed before the
change.

Cache and request shape:

  • The proxy adds and signs the Claude Code billing line only for models
    whose name starts with claude (reusing attributionHeaderPolicy). The
    line changed on every request, so OpenAI and GLM could only reuse the
    cached tool definitions. Measured on OpenRouter, later turns now hit the
    cache for about 99% of the input.
  • Responses requests carry the CLI session id as prompt_cache_key;
    requests with tools send parallel_tool_calls: true; tool schemas drop
    the $schema URL and zod's safe-integer bounds.
  • Cache hits are also read from prompt_cache_hit_tokens and a top-level
    cached_tokens. Responses report the requested model name.

Transport:

  • The proxy route turns off Bun's 60-second idle timeout per request.
  • A base URL that already ends in a version (/v4, /v1beta/openai) is
    used as the full prefix for requests, model lists and connection tests;
    others still get /v1.
  • The caller's abort signal reaches the upstream fetch, also before the
    upstream answers and for non-streaming requests.
  • With a manual HTTP proxy, the CLI env gets NO_PROXY for loopback, merged
    with the inherited value (an inherited * is kept as is). Scheduled
    tasks now get the network env too.
  • One stream supervisor times idle gaps (aiRequestTimeoutMs; only events
    sent to the CLI count, keep-alive comments become ping) and records how
    the stream ended. The CLI direct path gets the same limit through
    CLAUDE_ENABLE_STREAM_WATCHDOG and CLAUDE_STREAM_IDLE_TIMEOUT_MS.
  • Every upstream request is recorded as an upstream_request diagnostics
    event (provider, model, host, status, timings, request id, outcome);
    never the key or the body. diagnosticsService resolves its paths before
    awaiting, so a pending write cannot land in another config dir.
  • All three SSE readers share one spec framing function; a formatted JSON
    error body is surfaced instead of an empty reply.
  • OpenAI-format providers can carry extraHeaders and queryParams;
    configured headers go first, auth and Content-Type override them. They
    are rejected for anthropic-format providers. No desktop form yet.
  • The proxy answers count_tokens and other unknown paths with an Anthropic
    not_found_error. The server marks proxied providers with
    DREAMCODER_PROVIDER_PROXIED, and /context uses the local estimate for
    them instead of a Haiku request.

Docs: CHANGELOG entries, status lines on the fixed items in
docs/features/harness/proxy, measured cache results in the harness README.

This PR also carries the two research documents the issue builds on:
docs/features/harness comparisons of the agent runtime and the proxy
layer with Codex CLI and ZCode.

How I tested it

  • The changed test files plus diagnostics, cua and openaiAuth, run with
    bun test --randomize: 341 pass, 0 fail.
  • Full bun test: 2807 pass, 11 fail; main fails the same 11 locally
    (ripgrep not on PATH and other local environment causes).
  • desktop: bun run lint passes, bun run test -- --run 3104 pass.
  • Cache effect measured through sendToUpstream against OpenRouter with a
    test key: gpt-4.1-mini hits about 99% of the input from turn 2, glm-4.6
    hits 13312 of 13371 input tokens on turn 3.
  • No comments in added lines (the AGENTS.md grep finds none).

Part of #262. Each item below came with a test that failed before the
change.

Cache and request shape:
- The proxy adds and signs the Claude Code billing line only for models
  whose name starts with `claude` (reusing attributionHeaderPolicy). The
  line changed on every request, so OpenAI and GLM could only reuse the
  cached tool definitions. Measured on OpenRouter, later turns now hit the
  cache for about 99% of the input.
- Responses requests carry the CLI session id as `prompt_cache_key`;
  requests with tools send `parallel_tool_calls: true`; tool schemas drop
  the `$schema` URL and zod's safe-integer bounds.
- Cache hits are also read from `prompt_cache_hit_tokens` and a top-level
  `cached_tokens`. Responses report the requested model name.

Transport:
- The proxy route turns off Bun's 60-second idle timeout per request.
- A base URL that already ends in a version (`/v4`, `/v1beta/openai`) is
  used as the full prefix for requests, model lists and connection tests;
  others still get `/v1`.
- The caller's abort signal reaches the upstream fetch, also before the
  upstream answers and for non-streaming requests.
- With a manual HTTP proxy, the CLI env gets NO_PROXY for loopback, merged
  with the inherited value (an inherited `*` is kept as is). Scheduled
  tasks now get the network env too.
- One stream supervisor times idle gaps (aiRequestTimeoutMs; only events
  sent to the CLI count, keep-alive comments become `ping`) and records how
  the stream ended. The CLI direct path gets the same limit through
  CLAUDE_ENABLE_STREAM_WATCHDOG and CLAUDE_STREAM_IDLE_TIMEOUT_MS.
- Every upstream request is recorded as an `upstream_request` diagnostics
  event (provider, model, host, status, timings, request id, outcome);
  never the key or the body. diagnosticsService resolves its paths before
  awaiting, so a pending write cannot land in another config dir.
- All three SSE readers share one spec framing function; a formatted JSON
  error body is surfaced instead of an empty reply.
- OpenAI-format providers can carry extraHeaders and queryParams;
  configured headers go first, auth and Content-Type override them. They
  are rejected for anthropic-format providers. No desktop form yet.
- The proxy answers count_tokens and other unknown paths with an Anthropic
  not_found_error. The server marks proxied providers with
  DREAMCODER_PROVIDER_PROXIED, and /context uses the local estimate for
  them instead of a Haiku request.

Docs: CHANGELOG entries, status lines on the fixed items in
docs/features/harness/proxy, measured cache results in the harness README.
@GoDiao
GoDiao merged commit ce209f6 into main Nov 18, 2026
4 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant