Skip to content

fix(opencode): read cache write tokens from raw usage#36752

Open
lewislf wants to merge 14 commits into
anomalyco:devfrom
lewislf:cache-write-tokens
Open

fix(opencode): read cache write tokens from raw usage#36752
lewislf wants to merge 14 commits into
anomalyco:devfrom
lewislf:cache-write-tokens

Conversation

@lewislf

@lewislf lewislf commented Jul 13, 2026

Copy link
Copy Markdown

Issue for this PR

Closes #36749

Type of change

  • Bug fix
  • New feature
  • Refactor / code improvement
  • Documentation

What does this PR do?

Anthropic models served through an OpenAI-compatible gateway always recorded cache.write: 0, so cache writes were billed as plain input (1.0x instead of 1.25x) and cost came out understated.

The OpenAI usage shape has no cache-write field, so @ai-sdk/openai-compatible sets cacheWrite: void 0 and hands the provider payload through as LanguageModelUsage.raw. Gateways fronting Anthropic do report the count there (cache_creation_input_tokens), but usage() only read inputTokenDetails.cacheWriteTokens, and the fallbacks in Session.getUsage only cover native anthropic / vertex / bedrock metadata, so the value ended at 0.

This reads the count from raw when the SDK has nothing to offer. Native providers are unaffected: the normalized field still wins. Session.getUsage already derives non-cached input as inputTokens - cacheRead - cacheWrite, so the breakdown lines up once the write count is populated.

#18440 covered the same symptom and was closed by #22224, which made getUsage read usage.cacheWriteInputTokens. That fixed providers that populate the field, but @ai-sdk/openai-compatible never does, so gateways on that path stayed broken.

How did you verify your code works?

Ran a real session against a gateway fronting Bedrock Anthropic, first turn of the session, and read the tokens back from the message store.

Before:

{"total": 19428, "input": 19428, "output": 4, "cache": {"write": 0, "read": 0}}

After:

{"total": 19432, "input": 2, "output": 4, "cache": {"write": 19426, "read": 0}}

which matches what the gateway reported for that request (prompt_tokens: 19428, cache_creation_input_tokens: 19426).

Also ran bun typecheck and bun test test/session from packages/opencode: typecheck clean, 368 pass / 0 fail.

To confirm it yourself, point a provider with npm: "@ai-sdk/openai-compatible" at any gateway fronting an Anthropic model with caching on, send a prompt long enough to be cached, and check cache.write on the assistant message.

Screenshots / recordings

Not a UI change.

Checklist

  • I have tested my changes locally
  • I have not included unrelated changes in this PR

@lewislf

lewislf commented Jul 14, 2026

Copy link
Copy Markdown
Author

Heads up on the overlap with #36861. Both PRs touch the same cache-write fallback, but they fix different paths, and #36861 does not cover the case in #36749.

#36861 reads the counts out of ProviderMetadata. For a provider on npm: "@ai-sdk/openai-compatible", that object never carries them. In openai-compatible-chat-language-model.ts the metadata is built as { [metadataKey]: {} } plus whatever an optional metadataExtractor returns, and only acceptedPredictionTokens / rejectedPredictionTokens are written into it afterwards. opencode wires a metadataExtractor only on the github-copilot path, so for a generic openai-compatible provider metadataUsage() lands on an empty record and cache.write stays 0.

The count does survive, just not there: convertOpenAICompatibleChatUsage sets cacheWrite: undefined and passes the gateway payload through as usage.raw. That is what this PR reads.

I checked this rather than assumed it. Applying #36861 on its own and running the gateway scenario from #36749 against it (usage.raw.cache_creation_input_tokens: 19426, no provider metadata) still yields cacheWriteInputTokens: undefined, so cache.write is still 0. Its own 30 tests pass; the raw path is simply not covered.

So the two look complementary rather than duplicate. #36861 covers providers that expose cache counts through metadata and also adds cache read; this one covers the openai-compatible SDK path, where metadata has nothing to offer. They do collide textually, so whichever lands first, the other needs a rebase. Happy to rebase on top of #36861 if that one goes first.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

fix: cache write tokens always 0 for Anthropic models behind an OpenAI-compatible provider

1 participant