fix(opencode): read cache write tokens from raw usage#36752
Conversation
|
Heads up on the overlap with #36861. Both PRs touch the same cache-write fallback, but they fix different paths, and #36861 does not cover the case in #36749. #36861 reads the counts out of The count does survive, just not there: I checked this rather than assumed it. Applying #36861 on its own and running the gateway scenario from #36749 against it ( So the two look complementary rather than duplicate. #36861 covers providers that expose cache counts through metadata and also adds cache read; this one covers the openai-compatible SDK path, where metadata has nothing to offer. They do collide textually, so whichever lands first, the other needs a rebase. Happy to rebase on top of #36861 if that one goes first. |
Issue for this PR
Closes #36749
Type of change
What does this PR do?
Anthropic models served through an OpenAI-compatible gateway always recorded
cache.write: 0, so cache writes were billed as plain input (1.0x instead of 1.25x) and cost came out understated.The OpenAI usage shape has no cache-write field, so
@ai-sdk/openai-compatiblesetscacheWrite: void 0and hands the provider payload through asLanguageModelUsage.raw. Gateways fronting Anthropic do report the count there (cache_creation_input_tokens), butusage()only readinputTokenDetails.cacheWriteTokens, and the fallbacks inSession.getUsageonly cover nativeanthropic/vertex/bedrockmetadata, so the value ended at 0.This reads the count from
rawwhen the SDK has nothing to offer. Native providers are unaffected: the normalized field still wins.Session.getUsagealready derives non-cached input asinputTokens - cacheRead - cacheWrite, so the breakdown lines up once the write count is populated.#18440 covered the same symptom and was closed by #22224, which made
getUsagereadusage.cacheWriteInputTokens. That fixed providers that populate the field, but@ai-sdk/openai-compatiblenever does, so gateways on that path stayed broken.How did you verify your code works?
Ran a real session against a gateway fronting Bedrock Anthropic, first turn of the session, and read the tokens back from the message store.
Before:
{"total": 19428, "input": 19428, "output": 4, "cache": {"write": 0, "read": 0}}After:
{"total": 19432, "input": 2, "output": 4, "cache": {"write": 19426, "read": 0}}which matches what the gateway reported for that request (
prompt_tokens: 19428,cache_creation_input_tokens: 19426).Also ran
bun typecheckandbun test test/sessionfrompackages/opencode: typecheck clean, 368 pass / 0 fail.To confirm it yourself, point a provider with
npm: "@ai-sdk/openai-compatible"at any gateway fronting an Anthropic model with caching on, send a prompt long enough to be cached, and checkcache.writeon the assistant message.Screenshots / recordings
Not a UI change.
Checklist