Skip to content

feat(llm,agent,usage): add cache_ttl and track cache writes by lifetime - #243

Open
msmakouz wants to merge 1 commit into
masterfrom
feat/prompt-cache-ttl
Open

msmakouz wants to merge 1 commit into
masterfrom
feat/prompt-cache-ttl

Conversation

@msmakouz

@msmakouz msmakouz commented Nov 5, 2026

Copy link
Copy Markdown
Contributor

What was changed

cache_ttl sets the prompt cache lifetime, as a call option or as a route default in providers[].options.cache_ttl. Agents set it with agent_options.prompt_cache.ttl.
The Claude driver puts "5m" or "1h" on every cache breakpoint and rejects other values before sending the request; other drivers ignore the option. Cache writes are reported per lifetime in tokens.cache_write_by_ttl and stored in the new token_usage_cache_writes table.

local result = llm.generate(prompt, { model = "claude-sonnet-5-5", cache_ttl = "1h" })                                                                                        
-- result.tokens.cache_write_by_ttl = { ["1h"] = 4800 }                                                                                                                       

@msmakouz
msmakouz requested a review from wolfy-j November 5, 2026 12:59
@msmakouz msmakouz self-assigned this Nov 5, 2026
@msmakouz
msmakouz marked this pull request as draft November 5, 2026 13:00
@msmakouz
msmakouz marked this pull request as ready for review November 5, 2026 13:45
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant