diff --git a/README.md b/README.md index 976216c..4bb0c67 100644 --- a/README.md +++ b/README.md @@ -20,6 +20,11 @@ Provides the foundational building blocks for developing concurrent AI agent app including actor model primitives, LLM integration, HTTP services, data storage, security, and configuration management — enabling developers to build sophisticated multiagent systems through Wippy's configuration-driven architecture. +Stored conversations with developer messages render a different prefix once after +upgrading on Claude and Bedrock. Their prompt cache restarts, and thinking blocks +bound to the old prefix no longer match. See the [LLM replay guidance](src/llm/src/README.md#positional-notes) +for handling retained thinking blocks across the upgrade. + ## Modules ### Framework Modules diff --git a/src/agent/src/README.md b/src/agent/src/README.md index 2deb9ce..1ac75ee 100644 --- a/src/agent/src/README.md +++ b/src/agent/src/README.md @@ -314,9 +314,10 @@ from the persisted notes. `prefix_fingerprint` is a sha256 over the system text and tools as drivers send them: the compiled system prompt together with ordinary (non-positional) system messages in the conversation, which Claude and Bedrock hoist into the top-level -system, and each tool's name, description, schema and native definition -(`meta.claude`). Registry ids and registry metadata (comment, tags, title) do -not reach the provider and are not part of it. It is stable across steps while +system, without blank text, and each tool's name, description, schema and native +definition (a `meta.claude` that names a `type`). Registry ids and other registry +metadata (comment, tags, title, a `meta.claude` without a `type`) do not reach the +provider and are not part of it. It is stable across steps while the sent system text and tools stay the same. It describes the request actually sent: when a checkpoint callback replaces the history, the fingerprint is recomputed from the replacement, and stale thinking in the replacement is judged against it. @@ -339,7 +340,9 @@ thinking blocks itself: 1. The host stores `response.prefix_fingerprint` as `metadata.prefix_fingerprint` on every row it persists from that step that carries `thinking_blocks` (the assistant row; function call messages built through `add_function_call` carry - no metadata, so blocks for a tool round live on its assistant row). + no metadata, so blocks for a tool round live on its assistant row). It stores + `response.metadata.content_positions` on the same row when present, which keeps + a response that interleaves thinking, text and calls in its order on replay. 2. On later steps the host passes `drop_stale_thinking = true`. Before the request, the agent removes the blocks of every message up to and including the last one whose `metadata.prefix_fingerprint` differs from the current @@ -352,11 +355,20 @@ The current tool round (the last assistant turn, while only its tool results, notes and user messages joining the tool-result turn follow it) keeps its blocks even when they are stale: the provider requires the final assistant turn of a tool round to keep its thinking, so those blocks are left to the provider's -drop-on-mismatch handling, and only earlier turns are stripped. User content +drop-on-mismatch handling, and only earlier turns are stripped. The agent marks +them `metadata.stale_thinking` on the copies it sends: when the route's final +thinking object cannot carry a binding (`thinking_effort` 0 on a route not declared +`always`, a `none` route, `between_tools`), Claude and Bedrock leave them out of the +request instead of sending them unbound, which an enforcing model rejects on every +retry. User content (a user message, a developer message or a positional system note) after an assistant turn with no tool result in between ends that turn; ordinary system messages, which drivers hoist into the top-level system, do not. +A truncated step (`truncation_reason = "tool_calls"`) discards its calls, so its +`metadata` drops them from `content_positions` along with the thinking that led to +them. Rows persisted from it replay as a turn that ends in text, or as nothing. + Stripping works on copies: the host's message tables keep their blocks, and the stored fingerprint never reaches the provider. When a stale turn still has a function call without a result, the step fails with that error and sends no @@ -415,8 +427,8 @@ The note ends with "Use this temporal context when relevant to your responses." empty or unknown timezone or a `format` that is not a non-empty string. `format` is an optional Go time layout for the date and time part. The note carries the exact minute and is never rebuilt; the framework does not inject it per step. -The removed `time_interval` setting fails compilation with an error naming -`turn_context`. +The legacy `time_interval` setting is ignored, so existing configurations still +compile. A malformed `time_format` still fails compilation. ### Token Tracking @@ -479,6 +491,8 @@ end start are delivered as soon as the failure is known. - Delivery happens before `after_execute` wrappers run; the wrappers and the return value still receive the complete batch keyed by call ID. +- The listener runs synchronously. Stop is observed only after a listener in + progress returns, so keep listeners short. - A listener fails by raising or by returning a non-nil second value. The batch then stops: no further call starts, calls still running are cancelled, results of calls that finished but were not yet delivered are dropped without @@ -658,7 +672,7 @@ return { execute = execute } ### Built-in Traits -- `wippy.agent.traits:time_aware` -- Declares `agent_options.turn_context.time` through `build_func_id`; the host renders the note with `wippy.agent:turn_context`. Configurable via context: `timezone` (default UTC), `time_format` (optional Go time layout). It contributes no prompt text; a `time_interval` setting is rejected. +- `wippy.agent.traits:time_aware` -- Declares `agent_options.turn_context.time` through `build_func_id`; the host renders the note with `wippy.agent:turn_context`. Configurable via context: `timezone` (default UTC), `time_format` (optional Go time layout). It contributes no prompt text; a legacy `time_interval` setting is ignored. A trait build function that returns an error fails `compiler.compile` with an error naming the trait and the function. diff --git a/src/agent/src/_index.yaml b/src/agent/src/_index.yaml index 1a10c82..cad77b9 100644 --- a/src/agent/src/_index.yaml +++ b/src/agent/src/_index.yaml @@ -55,6 +55,7 @@ entries: prompt: wippy.llm:prompt llm: wippy.llm:llm output: wippy.llm:output + thinking: wippy.llm.util:thinking canonical: wippy.agent:canonical # wippy.agent:canonical diff --git a/src/agent/src/agent.lua b/src/agent/src/agent.lua index 9e0c583..842609b 100644 --- a/src/agent/src/agent.lua +++ b/src/agent/src/agent.lua @@ -5,6 +5,7 @@ local funcs = require("funcs") local output = require("output") local hash = require("hash") local canonical = require("canonical") +local thinking = require("thinking") type UnifiedTool = { name: string, @@ -447,31 +448,39 @@ local function has_tools_with_schemas(tools: any): boolean end -- What providers bind cached prefixes and thinking blocks to, as drivers send --- it: the system text (the compiled prompt plus conversation system messages, --- which Claude and Bedrock hoist into the top-level system) and each tool's --- name, description, schema and native definition. Registry ids and registry --- metadata never reach the provider. The model is left out: providers drop --- blocks a different model cannot read on their own. +-- it: the visible system text (the compiled prompt plus conversation system +-- messages, which Claude and Bedrock hoist into the top-level system) and each +-- tool's name, description, schema and native definition. A meta.claude is a +-- native definition only when it names a type; registry ids and other registry +-- metadata never reach the provider. The model is left out: providers drop blocks +-- a different model cannot read on their own. local function prefix_projection(messages: {any}, tools: {any}?): table local system = {} for _, message in ipairs(messages) do if message.role == prompt.ROLE.SYSTEM and not prompt.is_positional_note(message) then if type(message.content) == "string" then - table.insert(system, message.content) + if not prompt.is_blank(message.content) then + table.insert(system, message.content) + end elseif type(message.content) == "table" then for _, part in ipairs(message.content) do - table.insert(system, part.type == "text" and part.text or part) + if part.type ~= "text" then + table.insert(system, part) + elseif not prompt.is_blank(part.text) then + table.insert(system, part.text) + end end end end end local projected_tools = {} for _, tool in ipairs(tools or {}) do + local native = type(tool.meta) == "table" and tool.meta.claude or nil table.insert(projected_tools, { name = tool.name, description = tool.description, schema = tool.schema, - native = type(tool.meta) == "table" and tool.meta.claude or nil, + native = type(native) == "table" and native.type and native or nil, }) end return { system = system, tools = projected_tools } @@ -674,16 +683,25 @@ function agent:step(prompt_builder: any, runtime_options: any): (table?, string? end local is_current = runtime_options.thinking_is_current local working = prompt.new(messages) - local _, strip_err = working:drop_stale_thinking(function(msg) - if is_current then - return is_current(msg, fingerprint) == true + -- Stripping and marking read the same verdict, so a host predicate decides + -- each message once. + local verdicts = {} + local function current(msg) + if verdicts[msg] == nil then + if is_current then + verdicts[msg] = is_current(msg, fingerprint) == true + else + verdicts[msg] = msg.metadata.prefix_fingerprint == fingerprint + end end - return msg.metadata.prefix_fingerprint == fingerprint - end) + return verdicts[msg] + end + local _, strip_err = working:drop_stale_thinking(current) if strip_err then return nil, output.errors.generate():kind(output.ERROR_TYPE.INVALID_REQUEST) :message(tostring(strip_err)):build() end + working:mark_stale_thinking(current) return working:get_messages(), nil end @@ -791,6 +809,9 @@ function agent:step(prompt_builder: any, runtime_options: any): (table?, string? -- to distinguish a tool retry from an empty no-tool response. response.truncation_reason = result.tool_calls and #result.tool_calls > 0 and "tool_calls" or "empty_output" + if response.truncation_reason == "tool_calls" and type(response.metadata) == "table" then + thinking.discard_calls(response.metadata, response.result) + end return response end diff --git a/src/agent/src/canonical.lua b/src/agent/src/canonical.lua index 1207488..76f39c9 100644 --- a/src/agent/src/canonical.lua +++ b/src/agent/src/canonical.lua @@ -1,13 +1,17 @@ local json = require("json") -- Serializes a value with object keys in sorted order, so equal values always --- produce equal text. Array indices are written as keys, which keeps sparse --- tables unambiguous. +-- produce equal text. Tables keep the array/object shape used by the wire encoder. local canonical = {} function canonical.encode(value: any): string if type(value) ~= "table" then return json.encode(value) or "null" end local keys, out = {}, {} + local encoded = json.encode(value) + if encoded and encoded:sub(1, 1) == "[" then + for _, child in ipairs(value) do out[#out+1] = canonical.encode(child) end + return "[" .. table.concat(out, ",") .. "]" + end for key in pairs(value) do keys[#keys+1] = tostring(key) end table.sort(keys) for _, key in ipairs(keys) do diff --git a/src/agent/src/tools/caller.lua b/src/agent/src/tools/caller.lua index af35922..02670ee 100644 --- a/src/agent/src/tools/caller.lua +++ b/src/agent/src/tools/caller.lua @@ -742,7 +742,11 @@ local function command_result(pending: PendingCall, ok: boolean?): any if err then return { result = nil, error = err, tool_call = pending.tool_call } end - return { result = payload and payload:data() or nil, error = nil, tool_call = pending.tool_call } + local result = nil + if payload then + result = payload:data() + end + return { result = result, error = nil, tool_call = pending.tool_call } end local _, err = command:result() return { @@ -771,20 +775,21 @@ local function cancel_pending(pending: {[any]: PendingCall}) end -- Records every pending call: responses that are already complete keep their --- real outcome, the rest are cancelled. +-- real outcome, the rest are cancelled before any listener runs. local function settle_after_stop(run: BatchRun, pending: {[any]: PendingCall}) + local settled = {} for call_id, call in pairs(pending) do - if run.error then break end pending[call_id] = nil local completed, ok = poll_response(call) - if completed then - record_result(run, call_id, command_result(call, ok)) - else - call.command:cancel() - record_result(run, call_id, cancelled_result(call.tool_call)) - end + if not completed then call.command:cancel() end + settled[call_id] = { call = call, completed = completed, ok = ok } + end + for call_id, outcome in pairs(settled) do + if run.error then break end + local entry = outcome.completed and command_result(outcome.call, outcome.ok) + or cancelled_result(outcome.call.tool_call) + record_result(run, call_id, entry) end - cancel_pending(pending) end -- Records pending calls in completion order. Returns true when cancellation was observed. @@ -883,7 +888,13 @@ local function execute_parallel(self: any, run: BatchRun, context: table?, valid end end - if entry then record_result(run, call_id, entry) end + if entry then + if cancelled or cancellation_requested(cancel) then + cancelled = true + settle_after_stop(run, pending) + end + if not run.error then record_result(run, call_id, entry) end + end if run.error then cancel_pending(pending) return diff --git a/src/agent/src/traits/init_time_aware.lua b/src/agent/src/traits/init_time_aware.lua index f979f0d..1fc3091 100644 --- a/src/agent/src/traits/init_time_aware.lua +++ b/src/agent/src/traits/init_time_aware.lua @@ -11,10 +11,6 @@ type TimeAwareResult = { -- Declares the per-turn time context; hosts render it with -- wippy.agent:turn_context and persist it with each user turn. local function execute(base_prompt: string?): (TimeAwareResult?, string?) - if ctx.get("time_interval") ~= nil then - return nil, "time_aware no longer rounds the time: remove time_interval; hosts render the exact minute " - .. "from agent_options.turn_context with wippy.agent:turn_context" - end -- The trait context merges with the agent context, so the layout key is -- namespaced: a generic "format" there belongs to someone else. local format = ctx.get("time_format") diff --git a/src/agent/test/_index.yaml b/src/agent/test/_index.yaml index 21ebb60..4d2ee82 100644 --- a/src/agent/test/_index.yaml +++ b/src/agent/test/_index.yaml @@ -99,6 +99,28 @@ entries: tool_caller: wippy.agent.tools:caller method: run_tests + # app:replay_safety_test + - name: replay_safety_test + kind: function.lua + meta: + name: Agent Replay Safety Test + type: test + group: Agent Framework + source: file://replay_safety_test.lua + modules: [json] + imports: + prompt: wippy.llm:prompt + agent: wippy.agent:agent + llm: wippy.llm:llm + claude_generate: wippy.llm.claude:generate + bedrock_generate: wippy.llm.bedrock:generate + claude_mapper: wippy.llm.claude:mapper + claude_client: wippy.llm.claude:client + bedrock_mapper: wippy.llm.bedrock:mapper + bedrock_client: wippy.llm.bedrock:client + test: wippy.test:test + method: run_tests + # app:context_test - name: context_test kind: function.lua @@ -374,6 +396,7 @@ entries: imports: test: wippy.test:test init_time_aware: wippy.agent.traits:init_time_aware + compiler: wippy.agent.compiler:compiler method: run_tests - name: turn_context_test diff --git a/src/agent/test/replay_safety_test.lua b/src/agent/test/replay_safety_test.lua new file mode 100644 index 0000000..0615943 --- /dev/null +++ b/src/agent/test/replay_safety_test.lua @@ -0,0 +1,499 @@ +local test = require("test") +local json = require("json") +local llm = require("llm") +local prompt = require("prompt") +local agent = require("agent") +local claude_generate = require("claude_generate") +local bedrock_generate = require("bedrock_generate") +local claude_mapper = require("claude_mapper") +local claude_client = require("claude_client") +local bedrock_mapper = require("bedrock_mapper") +local bedrock_client = require("bedrock_client") + +local BUDGET_POLICY = { thinking = "budget", default_max_tokens = 2000 } + +local function text(value) return { { type = "text", text = value } } end +local function blocks(name) return { { type = "thinking", thinking = "", signature = "sig-" .. name } } end + +local function install(route): any + local sent: any = {} + claude_generate._client = { ENDPOINTS = { MESSAGES = "/messages" }, request = function(_, payload, options) + table.insert(sent, { payload = payload, betas = options and options.beta_features }) + return { content = { { type = "text", text = "ok" } }, stop_reason = "end_turn", + usage = { input_tokens = 1, output_tokens = 1 }, metadata = {} } + end } + bedrock_generate._client = { converse = function(_, payload) + table.insert(sent, { payload = payload }) + return { output = { message = { role = "assistant", content = { { text = "ok" } } } }, stopReason = "end_turn", + usage = { inputTokens = 1, outputTokens = 1 }, metadata = {} } + end } + llm._usage_tracker = false + local card = { name = "m", id = "m", capabilities = {}, class = {}, max_tokens = 200000, output_tokens = 64000, providers = { route } } + llm._models = { get_by_name = function() return card, nil end, get_by_class = function() return {}, nil end } + llm._providers = { + driver_declares_legacy_reasoning_flag = function() return false end, + driver_output_policy = function() return BUDGET_POLICY, nil end, + open = function(id) + local handler = id:match("^bedrock") and bedrock_generate or claude_generate + return { generate = function(_, args) return handler.handler(args) end } + end, + } + return sent +end + +local function spec(id, effort, prompt_text) + return { id = id, name = id, description = id, model = "m", max_tokens = 4096, thinking_effort = effort, + prompt = prompt_text == nil and ("You are " .. id) or prompt_text, + tools = { lookup = { name = "lookup", description = "d", schema = { type = "object", properties = {} }, + registry_id = "app:lookup", context = {} } }, + memory = {}, prompt_funcs = {}, step_funcs = {}, tool_wrappers = {} } +end + +-- Agent A thinks and hands off inside a tool round; B runs the next step. +local function handoff_history() + local answer = { role = "assistant", content = text("handing off"), metadata = { + thinking_blocks = blocks("A"), prefix_fingerprint = "fingerprint-of-agent-a" } } + local builder = prompt.new({ { role = "user", content = text("q1") }, answer } :: any) + builder:add_function_call("handoff", '{"to":"b"}', "c1") + builder:add_function_result("handoff", "ok", "c1") + return builder, answer +end + +local function has(content, field) + for _, part in ipairs(content) do + if part[field] ~= nil then return true end + end + return false +end + +local function claude_thinking(content) + for _, part in ipairs(content) do + if part.type == "thinking" then return true end + end + return false +end + +local ROUTES = { + { driver = "Claude", id = "claude_route", between = { thinking = { type = "between_tools" } } }, + { driver = "Bedrock", id = "bedrock_route", + between = { additionalModelRequestFields = { thinking = { type = "between_tools" } } } }, +} + +local function carries_blocks(route, sent) + local turn = sent[1].payload.messages[2] + if route.driver == "Claude" then return claude_thinking(turn.content) end + return has(turn.content, "reasoningContent") +end + +local function sent_thinking(route, sent) + if route.driver == "Claude" then return sent[1].payload.thinking end + local fields = sent[1].payload.additionalModelRequestFields + return fields and fields.thinking +end + +local function claude_blocks(shape) + local out = {} + for _, item in ipairs(shape) do + if type(item) == "table" then + table.insert(out, { type = "tool_use", id = item[1], name = "lookup", input = { a = 1 } }) + elseif item:match("^s%d$") then + table.insert(out, { type = "thinking", thinking = "", signature = item }) + else + table.insert(out, { type = "text", text = item }) + end + end + return out +end + +local function bedrock_blocks(shape) + local out = {} + for _, item in ipairs(shape) do + if type(item) == "table" then + table.insert(out, { toolUse = { toolUseId = item[1], name = "lookup", input = { a = 1 } } }) + elseif item:match("^s%d$") then + table.insert(out, { reasoningContent = { reasoningText = { text = "", signature = item } } }) + else + table.insert(out, { text = item }) + end + end + return out +end + +local function claude_stream(shape) + local body = "" + local function add(name, data) body = body .. "event: " .. name .. "\ndata: " .. json.encode(data) .. "\n\n" end + add("message_start", { message = { usage = { input_tokens = 1 } } }) + for index, block in ipairs(claude_blocks(shape)) do + local at = index - 1 + if block.type == "thinking" then + add("content_block_start", { index = at, content_block = { type = "thinking", thinking = "", signature = "" } }) + add("content_block_delta", { index = at, delta = { type = "signature_delta", signature = block.signature } }) + elseif block.type == "text" then + add("content_block_start", { index = at, content_block = { type = "text", text = "" } }) + add("content_block_delta", { index = at, delta = { type = "text_delta", text = block.text } }) + else + add("content_block_start", { index = at, content_block = { type = "tool_use", id = block.id, name = block.name } }) + add("content_block_delta", { index = at, delta = { type = "input_json_delta", partial_json = '{"a":' } }) + end + add("content_block_stop", { index = at }) + end + add("message_delta", { delta = { stop_reason = "max_tokens" }, usage = { output_tokens = 2 } }) + add("message_stop", {}) + local done = false + return { stream = { read = function() + if done then return nil end + done = true + return body + end }, metadata = {} } +end + +local function bedrock_stream(shape) + local function header(name, value) + return string.char(#name) .. name .. string.char(7) .. string.pack(">I2", #value) .. value + end + local chunks = {} + local function add(name, data) + local headers = header(":event-type", name) .. header(":content-type", "application/json") + .. header(":message-type", "event") + local payload = json.encode(data) + local total = 12 + #headers + #payload + 4 + table.insert(chunks, string.pack(">I4I4I4", total, #headers, 0) .. headers .. payload .. string.pack(">I4", 0)) + end + add("messageStart", { role = "assistant" }) + for index, block in ipairs(bedrock_blocks(shape)) do + local at = index - 1 + if block.reasoningContent then + add("contentBlockDelta", { contentBlockIndex = at, delta = { reasoningContent = { + signature = block.reasoningContent.reasoningText.signature } } }) + elseif block.text then + add("contentBlockDelta", { contentBlockIndex = at, delta = { text = block.text } }) + else + add("contentBlockStart", { contentBlockIndex = at, start = { toolUse = { toolUseId = block.toolUse.toolUseId, name = "lookup" } } }) + add("contentBlockDelta", { contentBlockIndex = at, delta = { toolUse = { input = '{"a":' } } }) + end + add("contentBlockStop", { contentBlockIndex = at }) + end + add("messageStop", { stopReason = "max_tokens" }) + add("metadata", { usage = { inputTokens = 1, outputTokens = 2 } }) + local at = 0 + return { stream = { read = function() + at = at + 1 + return chunks[at] + end }, metadata = {} } +end + +local function claude_replay(rows) + local sent = install({ id = "claude_route", provider_model = "wire" }) + assert(claude_generate.handler({ model = "m", messages = rows, options = { max_tokens = 100 } })) + return sent[1].payload.messages +end + +local function bedrock_replay(rows) + local sent = install({ id = "bedrock_route", provider_model = "wire" }) + assert(bedrock_generate.handler({ model = "m", messages = rows, options = { max_tokens = 100 } })) + return sent[1].payload.messages +end + +local function claude_order(message) + local order = {} + for _, part in ipairs(message.content) do + table.insert(order, part.signature or part.id or part.text or part.type) + end + return order +end + +local function bedrock_order(message) + local order = {} + for _, part in ipairs(message.content) do + local reasoning = part.reasoningContent and part.reasoningContent.reasoningText + table.insert(order, reasoning and reasoning.signature or (part.toolUse and part.toolUse.toolUseId) or part.text or "other") + end + return order +end + +local TRUNCATED_PATHS: any = { + { name = "Claude JSON", replay = claude_replay, order = claude_order, extract = function(shape) + return claude_mapper.format_success_response({ content = claude_blocks(shape), stop_reason = "max_tokens", + usage = { input_tokens = 1, output_tokens = 1 } }, "m", {}) + end }, + { name = "Claude stream", replay = claude_replay, order = claude_order, extract = function(shape) + local _, _, stream = claude_client.process_stream(claude_stream(shape), {}) + return claude_mapper.format_streaming_response(stream, {}, stream.usage, stream.finish_reason, stream.metadata) + end }, + { name = "Bedrock JSON", replay = bedrock_replay, order = bedrock_order, extract = function(shape) + return bedrock_mapper.format_success_response({ output = { message = { role = "assistant", content = bedrock_blocks(shape) } }, + stopReason = "max_tokens", usage = { inputTokens = 1, outputTokens = 1 } }, {}) + end }, + { name = "Bedrock stream", replay = bedrock_replay, order = bedrock_order, extract = function(shape) + local _, _, stream = bedrock_client.process_converse_stream(bedrock_stream(shape), {}) + return bedrock_mapper.format_streaming_response(stream, {}, stream.usage, stream.finish_reason, stream.metadata) + end }, +} + +-- The step of an agent on a length-limited response, then the request a host sends +-- after persisting what the step returned. +local function replay_after_truncation(path, shape) + local extracted = path.extract(shape) + agent._llm = { generate = function() + return { result = extracted.result.content, tool_calls = extracted.result.tool_calls, + finish_reason = extracted.finish_reason, metadata = extracted.metadata, tokens = {} } + end } + local builder = prompt.new() + builder:add_user("q") + local response = assert(agent.new(spec("agent_a", 0)):step(builder, {})) + agent._llm = nil + test.is_true(response.truncated) + test.is_nil(response.tool_calls) + local messages = path.replay({ + { role = "user", content = text("q") }, + { role = "assistant", content = text(response.result), metadata = { + thinking_blocks = response.metadata.thinking_blocks, + content_positions = response.metadata.content_positions, + prefix_fingerprint = response.prefix_fingerprint } }, + { role = "user", content = text("try again") }, + }) + for _, message in ipairs(messages) do + if message.role == "assistant" then return path.order(message), response end + end + return {}, response +end + +local function define_tests() + describe("Agent replay safety", function() + after_each(function() + llm._providers = nil + llm._models = nil + llm._usage_tracker = nil + claude_generate._client = nil + bedrock_generate._client = nil + agent._llm = nil + end) + + it("keeps two cache markers across five steps with one builder on Bedrock and Claude", function() + for _, route in ipairs({ ROUTES[2], ROUTES[1] }) do + local sent = install({ id = route.id, provider_model = "wire" }) + local runner = assert(agent.new(spec("agent_a", 0))) + local builder = prompt.new() + local counts, unchanged = {}, {} + for step = 1, 5 do + if step == 3 then + builder:add_function_call("lookup", "{}", "call-1") + builder:add_function_result("lookup", "Found", "call-1") + else + builder:add_user("Question " .. step) + end + local before = json.encode(builder.messages) + local response = assert(runner:step(builder, {})) + unchanged[step] = json.encode(builder.messages) == before + local count = 0 + local payload = sent[step].payload + for _, block in ipairs(payload.system or {}) do + if block.cache_control or block.cachePoint then count = count + 1 end + end + for _, message in ipairs(payload.messages) do + for _, block in ipairs(message.content) do + if block.cache_control or block.cachePoint then count = count + 1 end + end + end + counts[step] = count + builder:add_assistant(response.result) + end + test.eq(table.concat(counts, ","), "2,2,2,2,2", route.driver .. " per-step cache markers") + for step = 1, 5 do + test.is_true(unchanged[step], route.driver .. " leaves builder unchanged at step " .. step) + end + end + end) + + for _, route in ipairs(ROUTES) do + local UNBINDABLE = { + { name = "default budget", thinking = nil, effort = 0 }, + { name = "adaptive at effort 0", thinking = "adaptive", effort = 0 }, + { name = "none", thinking = "none", effort = 0 }, + { name = "none with a positive effort", thinking = "none", effort = 40 }, + } + for _, case in ipairs(UNBINDABLE) do + it("sends no stale block on " .. route.driver .. ": " .. case.name, function() + local sent = install({ id = route.id, provider_model = "wire", thinking = case.thinking }) + local builder, answer = handoff_history() + assert(agent.new(spec("agent_b", case.effort)):step(builder, { drop_stale_thinking = true })) + test.is_false(carries_blocks(route, sent)) + test.is_nil(sent_thinking(route, sent)) + test.eq(answer.metadata.thinking_blocks[1].signature, "sig-A") + test.is_nil(answer.metadata.stale_thinking) + end) + end + + it("sends no stale block on " .. route.driver .. ": between_tools", function() + local sent = install({ id = route.id, provider_model = "wire", thinking = "adaptive", + options = { request = route.between } }) + local builder = handoff_history() + assert(agent.new(spec("agent_b", 30)):step(builder, { drop_stale_thinking = true })) + test.is_false(carries_blocks(route, sent)) + test.eq(sent_thinking(route, sent).type, "between_tools") + end) + + local BINDABLE = { + { name = "always at effort 0", thinking = "always", effort = 0 }, + { name = "adaptive above effort 0", thinking = "adaptive", effort = 30 }, + } + for _, case in ipairs(BINDABLE) do + it("keeps the stale block bound on " .. route.driver .. ": " .. case.name, function() + local sent = install({ id = route.id, provider_model = "wire", thinking = case.thinking }) + local builder = handoff_history() + assert(agent.new(spec("agent_b", case.effort)):step(builder, { drop_stale_thinking = true })) + test.is_true(carries_blocks(route, sent)) + test.eq(sent_thinking(route, sent).block_binding.prefix_mismatch_behavior, "drop_block") + end) + end + + it("keeps a current block on an effort-0 " .. route.driver .. " step", function() + local sent = install({ id = route.id, provider_model = "wire" }) + local probe = prompt.new() + probe:add_user("x") + local fingerprint = assert(agent.new(spec("agent_b", 0)):step(probe, {})).prefix_fingerprint + sent = install({ id = route.id, provider_model = "wire" }) + local builder, answer = handoff_history() + answer.metadata.prefix_fingerprint = fingerprint + assert(agent.new(spec("agent_b", 0)):step(builder, { drop_stale_thinking = true })) + test.is_true(carries_blocks(route, sent)) + end) + + it("sends no empty system block for an empty prompt on " .. route.driver, function() + for _, empty in ipairs({ "", " \n" }) do + local sent = install({ id = route.id, provider_model = "wire" }) + local builder = prompt.new() + builder:add_user("q") + assert(agent.new(spec("agent_b", 0, empty)):step(builder, {})) + test.is_nil(sent[1].payload.system) + end + end) + end + + for _, path in ipairs(TRUNCATED_PATHS) do + it("drops thinking after the first discarded call even before visible text: " .. path.name, function() + local order, response = replay_after_truncation(path, { "s1", { "c1" }, "s2", "A", { "c2" } }) + test.eq(table.concat(order, "|"), "s1|A") + test.eq(response.result, "A") + test.eq(#response.metadata.thinking_blocks, 1) + test.eq(response.metadata.thinking_blocks[1].signature, "s1") + test.eq(#response.metadata.content_positions, 1) + test.eq(response.metadata.content_positions[1].type, "text") + test.eq(response.metadata.content_positions[1].text, "A") + end) + + it("never replays thinking behind a discarded call: text, thinking, call: " .. path.name, function() + local order, response = replay_after_truncation(path, { "A", "s1", { "c1" } }) + test.eq(table.concat(order, "|"), "A") + test.eq(#response.metadata.thinking_blocks, 0) + for _, entry in ipairs(response.metadata.content_positions or {}) do + test.eq(entry.type, "text") + end + end) + + it("never replays thinking behind a discarded call: thinking, call: " .. path.name, function() + local order, response = replay_after_truncation(path, { "s1", { "c1" } }) + test.eq(table.concat(order, "|"), "") + test.eq(#response.metadata.thinking_blocks, 0) + end) + + it("keeps thinking in front of the kept text: " .. path.name, function() + local order = replay_after_truncation(path, { "s1", "A", "s2", { "c1" } }) + test.eq(table.concat(order, "|"), "s1|A") + end) + + it("keeps a plain-order response's thinking in front of its text: " .. path.name, function() + local order = replay_after_truncation(path, { "s1", "A", { "c1" } }) + test.eq(table.concat(order, "|"), "s1|A") + end) + end + + it("marks only what the host's predicate rejects", function() + local sent = install({ id = "claude_route", provider_model = "wire" }) + local builder = handoff_history() + assert(agent.new(spec("agent_b", 0)):step(builder, { + drop_stale_thinking = true, + thinking_is_current = function() return true end, + })) + test.is_true(claude_thinking(sent[1].payload.messages[2].content)) + end) + + it("marks nothing without drop_stale_thinking", function() + local sent = install({ id = "claude_route", provider_model = "wire" }) + local builder = handoff_history() + assert(agent.new(spec("agent_b", 0)):step(builder, {})) + test.is_true(claude_thinking(sent[1].payload.messages[2].content)) + end) + + describe("prefix fingerprint", function() + local function fingerprint(tool_meta, prompt_text) + install({ id = "claude_route", provider_model = "wire" }) + local compiled = spec("agent_b", 0, prompt_text) + compiled.tools.lookup.meta = tool_meta + local builder = prompt.new() + builder:add_user("q") + return assert(agent.new(compiled):step(builder, {})).prefix_fingerprint + end + + it("distinguishes schema arrays from objects while keeping equal schemas stable", function() + local function schema_fingerprint(schema) + local sent = install({ id = "claude_route", provider_model = "wire" }) + local compiled = spec("agent_b", 0) + compiled.tools.lookup.schema = schema + local builder = prompt.new() + builder:add_user("q") + local result = assert(agent.new(compiled):step(builder, {})) + return result.prefix_fingerprint, sent[1].payload.tools[1].input_schema + end + local array = { type = "object", properties = {}, enum = { "x" } } + local object = { type = "object", properties = {}, enum = { ["1"] = "x" } } + local array_fingerprint, array_wire = schema_fingerprint(array) + local object_fingerprint, object_wire = schema_fingerprint(object) + test.eq(json.encode(array_wire.enum), '["x"]') + test.eq(json.encode(object_wire.enum), '{"1":"x"}') + test.is_false(array_fingerprint == object_fingerprint, "different wire schemas need different fingerprints") + local reordered = {} + reordered.enum = { "x" } + reordered.properties = {} + reordered.type = "object" + test.eq(schema_fingerprint(reordered), array_fingerprint, "key insertion order is irrelevant") + test.eq(schema_fingerprint({ type = "object", properties = {}, enum = { "x" } }), array_fingerprint, + "equal schemas have equal fingerprints") + test.eq(schema_fingerprint({ enum = { ["1"] = "x" }, properties = {}, type = "object" }), object_fingerprint) + end) + + it("ignores meta.claude without a type, which never reaches the wire", function() + test.eq(fingerprint({ claude = { display_width_px = 10 } }), fingerprint({})) + end) + + it("ignores a meta.claude that is not a table", function() + test.eq(fingerprint({ claude = "native" }), fingerprint({})) + end) + + it("covers meta.claude with a type, which replaces the tool on the wire", function() + test.is_false(fingerprint({ claude = { type = "computer_20251124" } }) == fingerprint({})) + test.is_false(fingerprint({ claude = { type = "computer_20251124", display_width_px = 10 } }) + == fingerprint({ claude = { type = "computer_20251124", display_width_px = 20 } })) + end) + + it("ignores blank system text, which never reaches the wire", function() + test.eq(fingerprint({}, ""), fingerprint({}, " \n")) + end) + + it("ignores a blank conversation system message", function() + install({ id = "claude_route", provider_model = "wire" }) + local plain = prompt.new() + plain:add_user("q") + local with_blank = prompt.new({ + { role = "system", content = text(" ") }, + { role = "user", content = text("q") }, + }) + local compiled = agent.new(spec("agent_b", 0)) + test.eq(assert(compiled:step(with_blank, {})).prefix_fingerprint, + assert(compiled:step(plain, {})).prefix_fingerprint) + end) + end) + end) +end + +return test.run_cases(define_tests) diff --git a/src/agent/test/tools/caller_test.lua b/src/agent/test/tools/caller_test.lua index 4e46b56..48c7568 100644 --- a/src/agent/test/tools/caller_test.lua +++ b/src/agent/test/tools/caller_test.lua @@ -1222,6 +1222,39 @@ local function define_tests() return notes end + for _, observed in ipairs({ "returned", "listener" }) do + it("preserves scalar tool values in " .. observed .. " results across strategies", function() + for _, strategy in ipairs(STRATEGIES) do + tool_results["test:false"] = { result = false } + tool_results["test:zero"] = { result = 0 } + tool_results["test:empty"] = { result = "" } + local notes = recording_listener() + local caller = tool_caller.new():set_strategy(strategy):set_result_listener(notes.listener) + local returned, err = caller:execute({}, { + falsy = tool("falsy", "test:false"), + zero = tool("zero", "test:zero"), + empty = tool("empty", "test:empty"), + failed = tool("failed", "test:failing_tool"), + }) + test.is_nil(err) + test.eq(#notes.order, 4, strategy) + local results = returned + if observed == "listener" then results = notes.results end + for call_id in pairs(returned) do + test.eq(notes.counts[call_id], 1, strategy .. " " .. call_id) + end + test.is_nil(results.failed.result, strategy) + test.eq(results.failed.error, "Tool execution failed", strategy) + test.eq(results.zero.result, 0, strategy) + test.is_nil(results.zero.error, strategy) + test.eq(results.empty.result, "", strategy) + test.is_nil(results.empty.error, strategy) + test.is_nil(results.falsy.error, strategy) + test.eq(results.falsy.result, false, strategy .. " " .. observed .. " false result") + end + end) + end + it("accepts a function or nil and rejects other values", function() local caller = tool_caller.new() test.eq(caller:set_result_listener(function() end), caller) @@ -1361,6 +1394,63 @@ local function define_tests() test.eq(#cancelled_commands, 1) end) + for _, phase in ipairs({ "collection", "launch" }) do + it("cancels every pending call before a blocking stop listener during " .. phase, function() + local cancel = channel.new() + local listening, release, done = channel.new(1), channel.new(), channel.new(1) + local responses = { channel.new(1), channel.new(1) } + tool_results["test:pending_a"] = { response = responses[1], result = "committed a" } + tool_results["test:pending_b"] = { response = responses[2], execute = function() + cancel:close() + return "committed b" + end } + local calls = { tool("a", "test:pending_a"), tool("b", "test:pending_b") } + if phase == "launch" then table.insert(calls, tool("c", "test:calculator")) end + local notes = recording_listener() + local caller = tool_caller.new():set_strategy("parallel"):set_result_listener(function(call_id, result) + notes.listener(call_id, result) + if #notes.order == 1 then + listening:send(true) + release:receive() + end + end) + coroutine.spawn(function() + local results, err = caller:execute({}, calls, cancel) + done:send({ results = results, error = err }) + end) + local guard = time.timer("2s") + local first = channel.select({ listening:case_receive(), guard:channel():case_receive() }) + local cancelled_before_listener = #cancelled_commands + local effects = 0 + for index, registry_id in ipairs({ "test:pending_a", "test:pending_b" }) do + local cancelled = false + for _, id in ipairs(cancelled_commands) do + if id == registry_id then cancelled = true end + end + if not cancelled then + effects = effects + 1 + local response = assert(responses[index]) + response:send(true) + end + end + release:close() + local finished = channel.select({ done:case_receive(), guard:channel():case_receive() }) + guard:stop() + test.eq(first.channel, listening) + test.eq(finished.channel, done) + test.eq(cancelled_before_listener, 2, "all pending commands are cancelled before the first listener") + test.eq(effects, 0, "no pending tool commits while the listener blocks") + test.is_nil(finished.value.error) + test.eq(#notes.order, #calls) + for index = 1, #calls do + test.eq(notes.counts[index], 1, "listener fires once per call") + test.is_true(finished.value.results[index].cancelled) + test.is_nil(finished.value.results[index].result) + test.is_true(rawequal(notes.results[index], finished.value.results[index])) + end + end) + end + it("fails the batch on a listener error and stops the remaining calls", function() local failures = { returned = function() return nil, "store unavailable" end, diff --git a/src/agent/test/traits/init_time_aware_test.lua b/src/agent/test/traits/init_time_aware_test.lua index 9ce33e3..b1e59bf 100644 --- a/src/agent/test/traits/init_time_aware_test.lua +++ b/src/agent/test/traits/init_time_aware_test.lua @@ -1,5 +1,6 @@ local init_time_aware = require("init_time_aware") local funcs = require("funcs") +local compiler = require("compiler") local function define_tests() describe("Init Time Aware Trait", function() @@ -43,14 +44,32 @@ local function define_tests() end end) - it("rejects the removed time_interval setting and names its replacement", function() + it("ignores the legacy time_interval setting", function() local result, err = funcs.new() - :with_context({ timezone = "UTC", time_interval = 15 }) + :with_context({ timezone = "Europe/Berlin", time_format = "2006-01-02 15:04", time_interval = 15 }) :call("wippy.agent.traits:init_time_aware", "base prompt") - test.is_nil(result) - test.contains(tostring(err), "time_interval") - test.contains(tostring(err), "turn_context") + test.is_nil(err, "legacy time_interval is ignored") + test.not_nil(result) + test.is_nil(result.prompt) + test.eq(result.agent_options.turn_context.time.timezone, "Europe/Berlin") + test.eq(result.agent_options.turn_context.time.format, "2006-01-02 15:04") + end) + + it("compiles a time-aware agent with legacy time_interval context", function() + local result, err = compiler.compile({ + id = "test:legacy_time_aware", + model = "test-model", + prompt = "Base prompt", + context = { timezone = "Europe/Berlin", time_interval = 15 }, + traits = { "wippy.agent.traits:time_aware" }, + }) + + test.is_nil(err, "legacy time_interval does not fail compilation") + test.not_nil(result) + test.eq(result.prompt, "Base prompt") + test.eq(result.agent_options.turn_context.time.timezone, "Europe/Berlin") + test.is_nil(result.agent_options.turn_context.time.format) end) end) end diff --git a/src/llm/src/README.md b/src/llm/src/README.md index 517ddef..d0443af 100644 --- a/src/llm/src/README.md +++ b/src/llm/src/README.md @@ -117,6 +117,8 @@ Keys a driver builds structurally cannot be set this way; a `request` naming one A dotted entry reserves that nested field and every parent of it: a structured-output request may still set `text.verbosity` or `output_config.effort`, but not replace `text` or `output_config` with a value that is not an object. Generate leaves the structured-output fields open. +The output a request sends is decided once per route, before the card's `output_tokens` cap and context admission. Budget thinking (declared, or the Claude and Bedrock default at a positive effort) needs `max_tokens` above its budget, so a smaller or unset value rises to budget + 1024. The card cap then applies to the result and the budget shrinks to fit, keeping 1024 tokens for the answer where the 1024 minimum budget allows; a cap of 1024 or less sends no thinking. A changed caller value is reported as `metadata.adjusted.max_tokens = { requested, sent }` where `sent` is the wire value, a shrunk budget as `metadata.adjusted.thinking_budget`, and dropped thinking as `metadata.adjusted.thinking_effort`; `strict` rejects all three. Admission reserves that output, including the 2000 tokens Claude and Bedrock send for an unset `max_tokens`. A driver declares those two facts in its binding's `meta.output_policy` (`thinking`, `default_max_tokens`); without one, only the card cap on a caller value applies. + `request` is raw wire data: route facts, the card's `output_tokens` cap and `metadata.adjusted` describe the contract options, not the request. A route that declares `sampling: false` and sends `request.temperature` sends that temperature. Maps merge into the mapped body rather than replacing it, so a `request.thinking` on a budget route keeps the mapped `budget_tokens`; declare the matching route facts instead of overriding them through `request`. `request` applies to `generate` and `structured_output`; `embed`, `evaluate` and `status` do not send it. `request` belongs to the route: route `options.request` from the catalog always applies, while a caller `request` on a resolved call is rejected with `invalid_request`, like caller `accepts`. Direct `provider_id` calls may pass `request`: @@ -131,13 +133,15 @@ local response, err = llm.generate("Hello", { ### Thinking block binding -Claude and Bedrock store each response's `thinking` blocks as `{ type, thinking, signature }` and `redacted_thinking` blocks as `{ type, data }` in `metadata.thinking_blocks`, in response order, and replay every stored block on later turns in its position (Bedrock: as `reasoningContent.reasoningText` and `reasoningContent.redactedContent`). Claude replays each block unchanged except for fields an earlier stored shape carried that its type does not accept (`data` on `thinking`, `thinking` and `signature` on `redacted_thinking`). A Claude thinking block is bound to the conversation that produced it: the top-level system prompt, the tools, and every earlier message. After an agent handoff, a prompt or tool change, or a history edit, models that enforce the check reject a request that replays such a block with a 400. +Claude and Bedrock store each response's `thinking` blocks as `{ type, thinking, signature }` and `redacted_thinking` blocks as `{ type, data }` in `metadata.thinking_blocks`, in response order (Bedrock: replayed as `reasoningContent.reasoningText` and `reasoningContent.redactedContent`). A response that is thinking, then at most one text block, then tool calls replays in that order with no record. Any other response, such as thinking between text and tool calls, also carries `metadata.content_positions` and a `position` on each thinking block; a host that stores `content_positions` next to `thinking_blocks` on the assistant row (or the first function call row of the response when it has no text) gets the response order back, with each visible text block separate, as long as the calls of that response are consecutive function call rows, which consolidate into one assistant turn. A function result between those rows splits the turn, and the order is then not rebuilt across the split. Each emitted assistant turn drops trailing thinking, including turns assembled from rows without positions. Whitespace-only text blocks of a recorded response are not rendered: the API compares neither those nor the outer whitespace of a text block, so leaving them out changes no prefix. Rows without `content_positions`, including every row stored before it existed, replay as all the thinking blocks of the row, then its text as one block, then its calls; so do rows whose text, apart from outer whitespace, no longer equals the recorded text. A recorded call the conversation no longer holds leaves the layout, and the thinking behind the last remaining text or call is not replayed: it led to that call, and an assistant turn cannot end in it. `thinking.discard_calls` applies the same rule to a response's metadata for a caller that discards its calls. Positions never reach the wire, and other drivers ignore them. Claude replays each block unchanged except for fields an earlier stored shape carried that its type does not accept (`data` on `thinking`, `thinking` and `signature` on `redacted_thinking`). A Claude thinking block is bound to the conversation that produced it: the top-level system prompt, the tools, and every earlier message. After an agent handoff, a prompt or tool change, or a history edit, models that enforce the check reject a request that replays such a block with a 400. When a request replays at least one thinking block and sends a thinking object of type `adaptive` or `enabled` (a positive thinking_effort, or `thinking = always` at any effort), the driver sets `thinking.block_binding.prefix_mismatch_behavior = "drop_block"` and adds the `thinking-binding-controls-2026-08-01` beta. The API then drops the first mismatched block and every later thinking block instead of rejecting the request. The drop applies to that request only and the history keeps the blocks, so after one change to the system prompt, tools or an earlier message, every later request again drops every thinking block from the first one bound to the old prefix onward, until the history before the change no longer carries thinking blocks; such conversations continue without that earlier reasoning, and `metadata.input_transformations` shows the drops on every response. Setting `prefix_mismatch_behavior` also opts the request into enforcement: on models that enforce the check only for organizations created on or after 2026-08-31, an older organization's mismatched blocks reached the model before and are now dropped. A `block_binding` the thinking object already carries is kept as set, and any request whose thinking object carries `block_binding` sends the beta. The Claude driver sends one `anthropic-beta` header: the configured `beta_features`, then the betas of a configured `anthropic-beta` context header (any letter case), then request betas, each named once. Bedrock appends the beta to the `anthropic_beta` list of `additionalModelRequestFields` and `/input_transformations` to `additionalModelResponseFieldPaths`, each named once. Plain generation and structured output behave the same. The driver decides the binding after merging `options.request`, from the thinking object as sent: a `request.thinking.type = "between_tools"` gets no binding and no beta, and a `request.thinking.block_binding` is kept as set and sends the beta. A request without a thinking object (`adaptive` or `budget` at zero or absent effort, or `thinking = "none"`) carries no binding: `block_binding` exists only inside an adaptive or enabled thinking object, and the driver never turns thinking on to carry it, since on several models an omitted thinking object means no thinking. Such a request replays its blocks unbound, so keep that conversation append-only. `always` is the form that carries the binding at zero effort. -Claude Sonnet 5.5 `thinking.type = "between_tools"` rejects `block_binding`, so a between_tools request has no drop_block recovery. Routes that must survive agent handoffs or history edits use `thinking: always` with a low default effort (`options.thinking_effort: 10`, the `low` level; 20 through 50 map to `medium`); keep between_tools for append-only routes. between_tools also rejects the `xhigh` and `max` levels (thinking_effort 80 and above) and per-message effort changes. +A message whose `metadata.stale_thinking` is true carries blocks the caller knows are bound to another prefix (`mark_stale_thinking`, which the agent runner calls). Claude and Bedrock remove those blocks when the final thinking object, after `options.request` and for that route, cannot carry a binding, so the request succeeds and later requests do not repeat a rejection. An adaptive or enabled thinking object keeps them, bound with `drop_block`. Only the request changes; the stored history keeps its blocks. + +Claude Sonnet 5.5 `thinking.type = "between_tools"` rejects `block_binding`, so a between_tools request has no drop_block recovery for blocks nothing marks as stale. Routes that must survive agent handoffs or history edits use `thinking: always` with a low default effort (`options.thinking_effort: 10`, the `low` level; 20 through 50 map to `medium`); keep between_tools for append-only routes. between_tools also rejects the `xhigh` and `max` levels (thinking_effort 80 and above) and per-message effort changes. Responses to a bound request report each dropped block in `metadata.input_transformations`, for example `{ type = "thinking_dropped", path = "messages.1.content.0", reason = "prefix_binding_mismatch" }`. `prefix_binding_mismatch` means the conversation before the block changed; `model_binding_mismatch` means the conversation moved to a model that cannot read the block. Other values exist and later checks add more, for example the reason `organization_binding_mismatch` (a block from another organization) and the entry type `thinking_mismatch_allowed` (a block that failed the check but still reached the model); ignore entry types and reasons you do not recognize. An empty array means nothing was dropped. Streaming Claude responses report the array from `message_start`. After a mid-stream server-side fallback the API sends the final array again in the closing `message_delta`, in a location that is not documented; the driver does not capture that final array yet, so such a stream reports the array from `message_start`. Bedrock reports it from `messageStop`. @@ -248,7 +252,30 @@ builder:add_system("The current date and time is Tuesday, November 3, 2026 at 09 Developer messages, positional or not, render the same way on Claude and Bedrock: each text part becomes its own text block, `` + text + ``, after the content of the user turn it follows, or a new user turn at the start or after an assistant turn. The wrapper tells the model host guidance apart from the user's prose. Positional system notes render as plain text. Existing blocks are never edited, so the rendering does not depend on where cache markers sit. -Bedrock sends developer messages as user text instead of `system` blocks. A Bedrock conversation stored before this change therefore renders a different prefix once after upgrading: its cache restarts and its earlier thinking blocks no longer match. The provider rejects such blocks unless the request lets it drop them; hosts that keep thinking blocks across the upgrade strip them with `drop_stale_thinking`. +Claude developer messages render as separate wrapped user text blocks, and Bedrock sends them as user text instead of `system` blocks. A Claude or Bedrock conversation with developer messages stored before this change therefore renders a different prefix once after upgrading: its cache restarts and thinking blocks bound to the old prefix no longer match. The provider rejects such blocks unless the request lets it drop them; hosts that keep thinking blocks across the upgrade strip them with `drop_stale_thinking`. + +### Tool Call IDs + +History moves between providers, so a call id can be foreign to the provider a request goes to. Each driver sends ids it accepts unchanged and translates the others, on the call and its result alike, into `tool-` plus a sha256 digest of the id. The translation is the same on every request, because the id is part of the prefix, and the stored history keeps the original. + +| Driver | Limits on the wire | +|---|---| +| Claude | `[a-zA-Z0-9_-]+` | +| Bedrock | `[a-zA-Z0-9_-]+`, 64 characters | +| OpenAI (Responses) | 64 characters | + +OpenAI-compatible endpoints differ and send ids unchanged. Google matches results to calls by the provider's own id or by name. + +### Blank Content + +One rule decides what is blank: text with no non-whitespace character. + +- Blank text in user, system and developer messages never reaches a driver. A message left with no content goes entirely, and an image beside blank text stays. Tool results are sent as they are. +- An assistant turn, after consolidating its assistant and function call rows, with no visible text and no call renders as nothing on every driver, its thinking blocks included. Claude invents no text for it. Inside a turn that stays, whitespace-only assistant text is not rendered either, on any driver and whether or not the row has a recorded layout; text with something visible keeps its outer whitespace. + +A stored conversation with such a row renders a different prefix once after upgrading (Claude sent the text `intercepted`, Bedrock the thinking blocks alone, OpenAI-compatible an empty assistant message): its cache restarts from that row and later thinking blocks no longer match. A stored row with whitespace-only text beside a call also renders differently once after upgrading: its prompt cache restarts one time from that row, while thinking blocks stay valid, because the API does not compare whitespace-only text blocks. + +A request whose mapped conversation is empty fails with `invalid_request` before any HTTP request on Claude, Bedrock, Google and OpenAI. A system-role note as the only content stays a valid request on OpenAI, as does any system message on OpenAI-compatible. ### Thinking Blocks @@ -260,7 +287,9 @@ local builder, err = builder:drop_stale_thinking(function(msg) end) ``` -An empty `thinking_blocks` list counts as no blocks; messages without blocks are not consulted. The current round, the last assistant turn while the request continues it, is never stripped and never consulted: the provider requires the final assistant turn of a tool round to keep its thinking. The round continues while only its function results and user content that joins the tool-result turn follow it (a steering message, a developer or positional note, or the image message `get_messages` inserts after a tool result with `_images`). User content after the turn with no function result in between ends it: a user message, a developer message or a positional system note, which drivers render as a new user turn. Ordinary system messages, which drivers hoist into the top-level system, and cache markers do not. Stale blocks in the current round are left to the provider's own drop-on-mismatch handling; once a later assistant turn exists, or the turn is answered without tools and a user message follows, it is strippable like any other. An assistant turn is the run of consecutive assistant and function call messages. If a turn whose blocks would be removed still has a function call without a result (a function call message or a `function_call` content part), the call fails and leaves the history unchanged: the provider needs those blocks to continue the tool round. +`mark_stale_thinking` takes the same predicate and sets `metadata.stale_thinking` on copies of every message that carries blocks, up to and including the last one the predicate rejects. Call it after `drop_stale_thinking` to mark the current round's stale blocks, which that call keeps; the driver removes marked blocks when the request cannot bind them. + +An empty `thinking_blocks` list counts as no blocks; messages without blocks are not consulted. The current round, the last assistant turn while the request continues it, is never stripped and never consulted: the provider requires the final assistant turn of a tool round to keep its thinking. The round continues while only its function results and user content that joins the tool-result turn follow it (a steering message, a developer or positional note, or the image message `get_messages` inserts after a tool result with `_images`). User content after the turn with no function result in between ends it: a user message, a developer message or a positional system note, which drivers render as a new user turn. Ordinary system messages, which drivers hoist into the top-level system, and cache markers do not. Stale blocks in the current round are left to the provider's own drop-on-mismatch handling, or to the driver when they are marked and the request cannot bind them; once a later assistant turn exists, or the turn is answered without tools and a user message follows, it is strippable like any other. An assistant turn is the run of assistant and function call messages, with cache markers and hoisted system messages transparent to the turn. If a turn whose blocks would be removed still has a function call without a result (a function call message or a `function_call` content part), the call fails and leaves the history unchanged: the provider needs those blocks to continue the tool round. ### Builder Operations diff --git a/src/llm/src/_index.yaml b/src/llm/src/_index.yaml index 5790f6c..c9a96af 100644 --- a/src/llm/src/_index.yaml +++ b/src/llm/src/_index.yaml @@ -269,6 +269,7 @@ entries: "temperature": {"type": "number", "minimum": 0, "maximum": 2}, "max_tokens": {"type": "integer", "minimum": 1}, "thinking_effort": {"type": "integer", "minimum": 0, "maximum": 100}, + "thinking_budget_tokens": {"type": "integer", "minimum": 1024, "description": "Budget thinking tokens as planned by llm before the request; drivers called directly derive it from thinking_effort"}, "request": {"type": "object", "additionalProperties": true, "description": "Provider wire fields deep-merged into the request body after option mapping; request values win, maps merge, arrays replace. Keys the driver builds (messages, tools, ...) are rejected. Set on routes in the catalog; callers may set it only on direct provider_id calls"}, "tool_choice_fallback": {"type": "string", "enum": ["auto"], "description": "Permits sending a forced tool_choice as 'auto' to a model that cannot be forced; for callers that enforce tool use themselves. The response metadata.tool_choice reports the substitution"}, "model_profile": { @@ -561,6 +562,7 @@ entries: "temperature": {"type": "number", "minimum": 0, "maximum": 2}, "max_tokens": {"type": "integer", "minimum": 1}, "thinking_effort": {"type": "integer", "minimum": 0, "maximum": 100}, + "thinking_budget_tokens": {"type": "integer", "minimum": 1024, "description": "Budget thinking tokens as planned by llm before the request; drivers called directly derive it from thinking_effort"}, "request": {"type": "object", "additionalProperties": true, "description": "Provider wire fields deep-merged into the request body after option mapping; request values win, maps merge, arrays replace. Keys the driver builds (messages, tools, ...) are rejected. Set on routes in the catalog; callers may set it only on direct provider_id calls"}, "model_profile": { "type": "object", diff --git a/src/llm/src/bedrock/_index.yaml b/src/llm/src/bedrock/_index.yaml index 0cbea71..38c6d53 100644 --- a/src/llm/src/bedrock/_index.yaml +++ b/src/llm/src/bedrock/_index.yaml @@ -121,6 +121,7 @@ entries: imports: sigv4: wippy.llm.bedrock:sigv4 bedrock_credentials: wippy.llm.bedrock:credentials + thinking: wippy.llm.util:thinking transport: wippy.llm.util:transport - name: mapper @@ -135,6 +136,7 @@ entries: prompt: wippy.llm:prompt output: wippy.llm:output route: wippy.llm.util:route + tool_id: wippy.llm.util:tool_id transport: wippy.llm.util:transport - name: embed_titan @@ -240,6 +242,9 @@ entries: meta: name: AWS Bedrock LLM Driver comment: AWS Bedrock provider using Converse API for text generation and InvokeModel for embeddings + output_policy: + thinking: budget + default_max_tokens: 2000 tags: - bedrock - aws diff --git a/src/llm/src/bedrock/client.lua b/src/llm/src/bedrock/client.lua index 5c42a18..b0bb8b1 100644 --- a/src/llm/src/bedrock/client.lua +++ b/src/llm/src/bedrock/client.lua @@ -4,6 +4,7 @@ local base64 = require("base64") local env = require("env") local ctx = require("ctx") local transport = require("transport") +local thinking = require("thinking") local sigv4 = require("sigv4") local credentials = require("bedrock_credentials") @@ -416,6 +417,7 @@ function bedrock_client.process_converse_stream(stream_response, callbacks) local error_reported = false local metadata = stream_response.metadata or {} local redacted_bytes: {[number]: string} = {} + local layout = {} local function handle_event(event): string? -- Converse stream events use top-level keys instead of a "type" field @@ -434,6 +436,7 @@ function bedrock_client.process_converse_stream(stream_response, callbacks) content_blocks[index] = start_block if start_block.toolUse then + layout[index] = { kind = "tool_use", id = start_block.toolUse.toolUseId } tool_calls[index] = { id = start_block.toolUse.toolUseId or "", name = start_block.toolUse.name or "", @@ -448,6 +451,8 @@ function bedrock_client.process_converse_stream(stream_response, callbacks) if delta.text then local text_chunk = tostring(delta.text) full_content = full_content .. text_chunk + layout[index] = layout[index] or { kind = "text", text = "" } + layout[index].text = layout[index].text .. text_chunk on_content(text_chunk) elseif delta.reasoningContent then -- ConverseStream opens reasoning blocks with their first delta, not contentBlockStart. @@ -462,9 +467,11 @@ function bedrock_client.process_converse_stream(stream_response, callbacks) end thinking_blocks[index] = thinking_blocks[index] or { type = "redacted_thinking", data = "" } redacted_bytes[index] = (redacted_bytes[index] or "") .. bytes + layout[index] = { kind = "thinking", block = thinking_blocks[index] } else local block = thinking_blocks[index] or { type = "thinking", thinking = "", signature = "" } thinking_blocks[index] = block + layout[index] = { kind = "thinking", block = block } if rc.text then on_thinking(tostring(rc.text)) block.thinking = block.thinking .. rc.text @@ -536,6 +543,7 @@ function bedrock_client.process_converse_stream(stream_response, callbacks) content = full_content, tool_calls = final_tool_calls, thinking = transport.in_index_order(thinking_blocks), + content_positions = thinking.record_positions(transport.in_index_order(layout)), finish_reason = finish_reason, usage = usage, metadata = metadata diff --git a/src/llm/src/bedrock/generate.lua b/src/llm/src/bedrock/generate.lua index 317d166..5dfe234 100644 --- a/src/llm/src/bedrock/generate.lua +++ b/src/llm/src/bedrock/generate.lua @@ -99,6 +99,9 @@ function generate_handler.handler(contract_args) } local mapped = generate_handler._mapper.map_messages(contract_args.messages) + if #mapped.messages == 0 then + return nil, err:kind(output.ERROR_TYPE.INVALID_REQUEST):message(output.NO_TURN_MESSAGE):build() + end local inference_config, additional_fields, adjusted, options_err = generate_handler._mapper.map_options(contract_args.options or {}, contract_args.accepts) if options_err then return nil, err:kind(output.ERROR_TYPE.INVALID_REQUEST):message(options_err):build() @@ -109,7 +112,7 @@ function generate_handler.handler(contract_args) end if not inference_config.maxTokens then - inference_config.maxTokens = 2000 + inference_config.maxTokens = mapper.DEFAULT_MAX_TOKENS end local converse_payload = { @@ -159,7 +162,7 @@ function generate_handler.handler(contract_args) } -- The binding depends on the thinking object as sent, so it is the last change to the payload. - local bind_err = generate_handler._mapper.bind_thinking(converse_payload) + local bind_err = generate_handler._mapper.bind_thinking(converse_payload, mapped.stale_thinking) if bind_err then return nil, err:kind(output.ERROR_TYPE.INVALID_REQUEST):message(bind_err):build() end diff --git a/src/llm/src/bedrock/mapper.lua b/src/llm/src/bedrock/mapper.lua index 0115613..659bfa8 100644 --- a/src/llm/src/bedrock/mapper.lua +++ b/src/llm/src/bedrock/mapper.lua @@ -4,6 +4,7 @@ local prompt = require("prompt") local json = require("json") local route = require("route") local transport = require("transport") +local tool_ids = require("tool_id") local mapper = {} @@ -24,8 +25,15 @@ mapper.CAPABILITY = { -- Wire body keys the driver builds structurally; options.request cannot set them. mapper.REQUEST_KEYS = { "messages", "system", "toolConfig" } +-- Output sent for a request that leaves max_tokens unset; the driver binding's +-- meta.output_policy declares the same value so admission can reserve it. +mapper.DEFAULT_MAX_TOKENS = 2000 + local MAX_CACHE_CHECKPOINTS = 4 +-- ToolUseBlock.toolUseId and ToolResultBlock.toolUseId: at most 64 characters. +local MAX_TOOL_ID_LENGTH = 64 + -- Converse stopReason -> contract finish_reason mapper.FINISH_REASON_MAP = {} :: {[string]: string} mapper.FINISH_REASON_MAP["end_turn"] = output.FINISH_REASON.STOP @@ -192,7 +200,7 @@ local function last_cacheable_message_position(messages) local content: any = messages[message_index].content or {} for block_index = #content, 1, -1 do local block = content[block_index] - if block and ((block.text and block.text ~= "") or block.toolUse or block.toolResult + if block and ((block.text and not prompt.is_blank(block.text)) or block.toolUse or block.toolResult or block.image or block.document) then return { message = message_index, block = block_index } end @@ -240,18 +248,81 @@ local function reasoning_content(block) return nil end +-- Merges consecutive same-role messages into one turn. +local function consolidated_turns(messages) + local consolidated = {} + for _, msg in ipairs(messages) do + if #consolidated > 0 and consolidated[#consolidated].role == msg.role then + for _, block in ipairs(msg.content) do + table.insert(consolidated[#consolidated].content, block) + end + else + table.insert(consolidated, msg) + end + end + return consolidated +end + +-- An assistant turn with no visible text and no call renders as nothing, thinking +-- included: there is nothing in it to replay against. +local function without_blank_turns(messages) + local kept = {} + for _, msg in ipairs(messages) do + local visible = msg.role ~= "assistant" + for _, block in ipairs(msg.role == "assistant" and msg.content or {}) do + if not block.reasoningContent and not block.cachePoint then + visible = true + end + end + if visible then + table.insert(kept, msg) + end + end + return kept +end + -- Map contract messages to Converse format function mapper.map_messages(contract_messages) if not contract_messages or #contract_messages == 0 then return { messages = {}, system = nil } end + contract_messages = prompt.without_blank_text(contract_messages) + local order = thinking.new_turn_order() + local translate_id = tool_ids.mapper(contract_messages, + { pattern = tool_ids.NATIVE_PATTERN, max_length = MAX_TOOL_ID_LENGTH }) + local function tool_use_id(id) + if id == nil or id == "" then return "" end + return translate_id(id) + end + local stale_thinking = {} + + local function add_reasoning_blocks(content_blocks, msg) + if not (msg.metadata and msg.metadata.thinking_blocks) then + return + end + for _, thinking_block in ipairs(msg.metadata.thinking_blocks) do + local reasoning = order.keeps_thinking(thinking_block) and reasoning_content(thinking_block) + if reasoning then + order.place_thinking(reasoning, thinking_block) + if msg.metadata.stale_thinking then + stale_thinking[reasoning] = true + end + table.insert(content_blocks, reasoning) + end + end + end + local converse_messages = {} local system_blocks = {} local system_cache_positions = {} local message_cache_positions = {} for _, msg in ipairs(contract_messages) do + if msg.role ~= prompt.ROLE.ASSISTANT and msg.role ~= prompt.ROLE.FUNCTION_CALL + and not prompt.transparent_to_assistant_turn(msg) then + order.leave_turn() + end if msg.role == prompt.ROLE.SYSTEM and not prompt.is_positional_note(msg) then if type(msg.content) == "string" then table.insert(system_blocks, { text = msg.content }) @@ -285,7 +356,7 @@ function mapper.map_messages(contract_messages) content = { { toolResult = { - toolUseId = msg.function_call_id or "", + toolUseId = tool_use_id(msg.function_call_id), content = { { text = result_text } }, status = msg.is_error == true and "error" or "success" } @@ -296,22 +367,18 @@ function mapper.map_messages(contract_messages) local arguments = normalize_tool_arguments(msg.function_call.arguments) local content_blocks = {} - if msg.metadata and msg.metadata.thinking_blocks then - for _, thinking_block in ipairs(msg.metadata.thinking_blocks) do - local reasoning = reasoning_content(thinking_block) - if reasoning then - table.insert(content_blocks, reasoning) - end - end - end + order.enter_row(msg) + add_reasoning_blocks(content_blocks, msg) - table.insert(content_blocks, { + local tool_use = { toolUse = { - toolUseId = msg.function_call.id or "", + toolUseId = tool_use_id(msg.function_call.id), name = msg.function_call.name or "", input = arguments } - }) + } + order.place_call(tool_use, msg.function_call.id) + table.insert(content_blocks, tool_use) table.insert(converse_messages, { role = "assistant", @@ -320,33 +387,37 @@ function mapper.map_messages(contract_messages) elseif msg.role == prompt.ROLE.ASSISTANT then local content_blocks = {} - if msg.metadata and msg.metadata.thinking_blocks then - for _, thinking_block in ipairs(msg.metadata.thinking_blocks) do - local reasoning = reasoning_content(thinking_block) - if reasoning then - table.insert(content_blocks, reasoning) - end + local layout = order.enter_row(msg) + add_reasoning_blocks(content_blocks, msg) + + if layout then + for _, entry in ipairs(layout.texts) do + local text_block = { text = entry.text } + order.place_text(text_block, entry.position) + table.insert(content_blocks, text_block) end end local content = msg.content if type(content) == "string" then - if content ~= "" then + if not prompt.is_blank(content) and not layout then table.insert(content_blocks, { text = content }) end elseif type(content) == "table" then for _, part in ipairs(content) do - if part.type == "text" and part.text and part.text ~= "" then + if part.type == "text" and not prompt.is_blank(part.text) and not layout then table.insert(content_blocks, { text = part.text }) elseif part.type == "function_call" then local args = normalize_tool_arguments(part.arguments) - table.insert(content_blocks, { + local tool_use = { toolUse = { - toolUseId = part.id or "", + toolUseId = tool_use_id(part.id), name = part.name or "", input = args } - }) + } + order.place_call(tool_use, part.id) + table.insert(content_blocks, tool_use) elseif part.type == "image" then local img = convert_image_to_converse(part) if img then @@ -398,26 +469,25 @@ function mapper.map_messages(contract_messages) local message = (converse_messages :: any)[pos.message] local block_index = math.tointeger(pos.block) if message and block_index then - table.insert(message.content, block_index + 1, { cachePoint = { type = "default" } }) + local cache_point = { cachePoint = { type = "default" } } + order.place_behind(cache_point, message.content[block_index]) + table.insert(message.content, block_index + 1, cache_point) end end end - -- Consolidate consecutive same-role messages - local consolidated = {} - for _, msg in ipairs(converse_messages) do - if #consolidated > 0 and consolidated[#consolidated].role == msg.role then - for _, block in ipairs(msg.content) do - table.insert(consolidated[#consolidated].content, block) - end - else - table.insert(consolidated, msg) + local consolidated = consolidated_turns(converse_messages) + for _, msg in ipairs(consolidated) do + if msg.role == "assistant" then + msg.content = order.rebuild(msg.content) end end return { - messages = consolidated, - system = #system_blocks > 0 and system_blocks or nil + -- Converse takes alternating roles, so turns around a removed blank turn merge. + messages = consolidated_turns(without_blank_turns(consolidated)), + system = #system_blocks > 0 and system_blocks or nil, + stale_thinking = next(stale_thinking) and stale_thinking or nil } end @@ -511,8 +581,7 @@ function mapper.map_options(contract_options, accepts): (table, table, table, st additional_fields.output_config = { effort = thinking.effort_level(contract_options.thinking_effort) } end elseif mode == "budget" and contract_options.thinking_effort and contract_options.thinking_effort > 0 then - local thinking_budget = 1024 + (24000 - 1024) * (contract_options.thinking_effort / 100) - thinking_budget = math.floor(thinking_budget + 0.5) + local thinking_budget, max_tokens = route.budget_output(contract_options) additional_fields.thinking = { type = "enabled", @@ -526,10 +595,7 @@ function mapper.map_options(contract_options, accepts): (table, table, table, st inference_config.temperature = nil end - -- Ensure max_tokens accommodates thinking budget - if not inference_config.maxTokens or inference_config.maxTokens <= thinking_budget then - inference_config.maxTokens = thinking_budget + 1024 - end + inference_config.maxTokens = max_tokens end -- Pass through top_k as additional field @@ -556,9 +622,14 @@ end -- Binds the reasoning a Converse payload replays to drop_block. Bedrock takes the beta in -- the anthropic_beta model field and returns input_transformations only when the payload -- names it in additionalModelResponseFieldPaths. Returns an error when either list a --- bound payload extends is not a list. -function mapper.bind_thinking(payload): string? +-- bound payload extends is not a list. A thinking object that cannot carry a binding +-- sends none of the blocks in `stale_blocks` (from map_messages): unbound, the API +-- would reject them. +function mapper.bind_thinking(payload, stale_blocks): string? local fields = payload.additionalModelRequestFields + if not thinking.is_bindable(fields and fields.thinking) then + thinking.remove_blocks(payload.messages, stale_blocks) + end if not fields then return nil end @@ -661,25 +732,32 @@ function mapper.extract_response_content(converse_response, tool_names) end local text_blocks = {} + local entries = {} for _, block in ipairs(converse_response.output.message.content) do if block.text then content_text = content_text .. block.text table.insert(text_blocks, block.text) + table.insert(entries, { kind = "text", text = block.text }) elseif block.toolUse then + table.insert(entries, { kind = "tool_use", id = block.toolUse.toolUseId }) table.insert(tool_calls, { id = block.toolUse.toolUseId or "", name = block.toolUse.name or "", arguments = block.toolUse.input or {} }) elseif block.reasoningContent and block.reasoningContent.reasoningText then - table.insert(thinking_blocks, { + local stored = { type = "thinking", thinking = block.reasoningContent.reasoningText.text or "", signature = block.reasoningContent.reasoningText.signature or "" - }) + } + table.insert(entries, { kind = "thinking", block = stored }) + table.insert(thinking_blocks, stored) elseif block.reasoningContent and block.reasoningContent.redactedContent then - table.insert(thinking_blocks, { type = "redacted_thinking", data = block.reasoningContent.redactedContent }) + local stored = { type = "redacted_thinking", data = block.reasoningContent.redactedContent } + table.insert(entries, { kind = "thinking", block = stored }) + table.insert(thinking_blocks, stored) end end @@ -696,6 +774,7 @@ function mapper.extract_response_content(converse_response, tool_names) }) -- Clear the content_text since the "content" was actually the tool call content_text = "" + entries = {} break end end @@ -704,7 +783,8 @@ function mapper.extract_response_content(converse_response, tool_names) return { content = content_text, tool_calls = tool_calls, - thinking_blocks = thinking_blocks + thinking_blocks = thinking_blocks, + content_positions = thinking.record_positions(entries) } end @@ -743,8 +823,9 @@ function mapper.format_success_response(converse_response, name_to_id_map) tokens.thinking_tokens = approximate_token_count(thinking_content) end - -- If fallback parser recovered tool calls from text, treat it as tool_call stop - if #extracted.tool_calls > 0 and finish_reason ~= "tool_call" then + -- Tool calls make a tool_call stop (the fallback parser may have recovered them + -- from text), except a length stop: its last call can be incomplete. + if #extracted.tool_calls > 0 and finish_reason ~= "tool_call" and finish_reason ~= output.FINISH_REASON.LENGTH then finish_reason = "tool_call" end @@ -761,6 +842,7 @@ function mapper.format_success_response(converse_response, name_to_id_map) result.metadata.thinking = thinking_content result.metadata.thinking_blocks = extracted.thinking_blocks + result.metadata.content_positions = extracted.content_positions return result end @@ -789,6 +871,7 @@ function mapper.format_streaming_response(stream_result, name_to_id_map, usage, result.metadata.thinking = thinking_content result.metadata.thinking_blocks = thinking_blocks + result.metadata.content_positions = stream_result.content_positions return result end diff --git a/src/llm/src/bedrock/structured_output.lua b/src/llm/src/bedrock/structured_output.lua index b30bf3d..e73a822 100644 --- a/src/llm/src/bedrock/structured_output.lua +++ b/src/llm/src/bedrock/structured_output.lua @@ -92,6 +92,9 @@ function structured_output_handler.handler(contract_args) end local mapped = structured_output_handler._mapper.map_messages(contract_args.messages) + if #mapped.messages == 0 then + return nil, err:kind(output.ERROR_TYPE.INVALID_REQUEST):message(output.NO_TURN_MESSAGE):build() + end local inference_config, additional_fields, adjusted, options_err = structured_output_handler._mapper.map_options(contract_args.options or {}, contract_args.accepts) if options_err then return nil, err:kind(output.ERROR_TYPE.INVALID_REQUEST):message(options_err):build() @@ -102,7 +105,7 @@ function structured_output_handler.handler(contract_args) end if not inference_config.maxTokens then - inference_config.maxTokens = 2000 + inference_config.maxTokens = mapper.DEFAULT_MAX_TOKENS end -- Build tool for structured output extraction @@ -144,7 +147,7 @@ function structured_output_handler.handler(contract_args) end -- The binding depends on the thinking object as sent, so it is the last change to the payload. - local bind_err = structured_output_handler._mapper.bind_thinking(converse_payload) + local bind_err = structured_output_handler._mapper.bind_thinking(converse_payload, mapped.stale_thinking) if bind_err then return nil, err:kind(output.ERROR_TYPE.INVALID_REQUEST):message(bind_err):build() end diff --git a/src/llm/src/claude/_index.yaml b/src/llm/src/claude/_index.yaml index 2b28883..67d066f 100644 --- a/src/llm/src/claude/_index.yaml +++ b/src/llm/src/claude/_index.yaml @@ -40,6 +40,7 @@ entries: imports: mapper: wippy.llm.claude:mapper output: wippy.llm:output + thinking: wippy.llm.util:thinking transport: wippy.llm.util:transport # wippy.llm.claude:driver @@ -48,6 +49,9 @@ entries: meta: name: Claude LLM Driver comment: Claude implementation of LLM contracts with context-based configuration and streaming support + output_policy: + thinking: budget + default_max_tokens: 2000 tags: - claude - llm @@ -112,12 +116,12 @@ entries: source: file://mapper.lua modules: - json - - hash imports: thinking: wippy.llm.util:thinking prompt: wippy.llm:prompt output: wippy.llm:output route: wippy.llm.util:route + tool_id: wippy.llm.util:tool_id # wippy.llm.claude:provider - name: provider diff --git a/src/llm/src/claude/client.lua b/src/llm/src/claude/client.lua index cb84c35..0e09b33 100644 --- a/src/llm/src/claude/client.lua +++ b/src/llm/src/claude/client.lua @@ -3,6 +3,7 @@ local http_client = require("http_client") local env = require("env") local ctx = require("ctx") local transport = require("transport") +local thinking = require("thinking") type ClaudeConfig = { api_key: string?, @@ -223,7 +224,7 @@ function claude_client.request(endpoint_path, payload, options): (any, transport return parsed end -local function stream_result(full_content, tool_calls, thinking_blocks, finish_reason, usage, metadata) +local function stream_result(full_content, tool_calls, thinking_blocks, layout, finish_reason, usage, metadata) local final_tool_calls = {} for _, tool_call in ipairs(transport.in_index_order(tool_calls)) do if type(tool_call) == "table" and tool_call.id then @@ -235,6 +236,7 @@ local function stream_result(full_content, tool_calls, thinking_blocks, finish_r content = full_content, tool_calls = final_tool_calls, thinking = transport.in_index_order(thinking_blocks), + content_positions = thinking.record_positions(transport.in_index_order(layout)), finish_reason = finish_reason, usage = usage, metadata = metadata @@ -259,6 +261,7 @@ function claude_client.process_stream(stream_response, callbacks) local finish_reason = nil local usage = {} local content_blocks = {} + local layout = {} local leftover = "" local metadata = stream_response.metadata or {} @@ -331,6 +334,11 @@ function claude_client.process_stream(stream_response, callbacks) data = data.content_block.data } end + if thinking_blocks[data.index] then + layout[data.index] = { kind = "thinking", block = thinking_blocks[data.index] } + elseif data.content_block.type == "tool_use" then + layout[data.index] = { kind = "tool_use", id = data.content_block.id } + end end elseif event_type == "content_block_delta" then local index = data.index or 0 @@ -339,6 +347,8 @@ function claude_client.process_stream(stream_response, callbacks) if delta.type == "text_delta" then local text_chunk = delta.text or "" full_content = full_content .. text_chunk + layout[index] = layout[index] or { kind = "text", text = "" } + layout[index].text = layout[index].text .. text_chunk on_content(text_chunk) elseif delta.type == "thinking_delta" then local thinking_chunk = delta.thinking or "" @@ -395,7 +405,7 @@ function claude_client.process_stream(stream_response, callbacks) end end elseif event_type == "message_stop" then - local result = stream_result(full_content, tool_calls, thinking_blocks, finish_reason, usage, metadata) + local result = stream_result(full_content, tool_calls, thinking_blocks, layout, finish_reason, usage, metadata) on_done(result) return full_content, nil, result elseif event_type == "error" then @@ -411,7 +421,7 @@ function claude_client.process_stream(stream_response, callbacks) ::continue:: end - local result = stream_result(full_content, tool_calls, thinking_blocks, finish_reason, usage, metadata) + local result = stream_result(full_content, tool_calls, thinking_blocks, layout, finish_reason, usage, metadata) on_done(result) return full_content, nil, result end diff --git a/src/llm/src/claude/generate.lua b/src/llm/src/claude/generate.lua index 71c36e5..81ac6e5 100644 --- a/src/llm/src/claude/generate.lua +++ b/src/llm/src/claude/generate.lua @@ -101,6 +101,9 @@ function generate_handler.handler(contract_args) } local mapped_messages = generate_handler._mapper.map_messages(contract_args.messages) + if #mapped_messages.messages == 0 then + return nil, err:kind(output.ERROR_TYPE.INVALID_REQUEST):message(output.NO_TURN_MESSAGE):build() + end local mapped_options, adjusted, options_err = generate_handler._mapper.map_options(contract_args.options or {}, contract_args.accepts) if options_err then return nil, err:kind(output.ERROR_TYPE.INVALID_REQUEST):message(options_err):build() @@ -113,7 +116,7 @@ function generate_handler.handler(contract_args) local claude_payload = { model = contract_args.model, messages = mapped_messages.messages, - max_tokens = mapped_options.max_tokens or 2000 + max_tokens = mapped_options.max_tokens or mapper.DEFAULT_MAX_TOKENS } if mapped_messages.system then @@ -165,7 +168,7 @@ function generate_handler.handler(contract_args) end -- The binding depends on the thinking object as sent, so it is the last change to the payload. - request_options.beta_features = generate_handler._mapper.bind_thinking(claude_payload) + request_options.beta_features = generate_handler._mapper.bind_thinking(claude_payload, mapped_messages.stale_thinking) local response, request_err = generate_handler._client.request( generate_handler._client.ENDPOINTS.MESSAGES, diff --git a/src/llm/src/claude/mapper.lua b/src/llm/src/claude/mapper.lua index 8601a0f..bbb1154 100644 --- a/src/llm/src/claude/mapper.lua +++ b/src/llm/src/claude/mapper.lua @@ -2,7 +2,7 @@ local thinking = require("thinking") local output = require("output") local prompt = require("prompt") local json = require("json") -local hash = require("hash") +local tool_ids = require("tool_id") local route = require("route") local mapper = {} @@ -55,6 +55,10 @@ mapper.HTTP_STATUS_MAP[502] = output.ERROR_TYPE.SERVER_ERROR mapper.HTTP_STATUS_MAP[503] = output.ERROR_TYPE.SERVER_ERROR mapper.HTTP_STATUS_MAP[529] = output.ERROR_TYPE.SERVER_ERROR +-- Output sent for a request that leaves max_tokens unset; the driver binding's +-- meta.output_policy declares the same value so admission can reserve it. +mapper.DEFAULT_MAX_TOKENS = 2000 + -- Claude API cache control limit local MAX_CACHE_BREAKPOINTS = 4 @@ -64,55 +68,13 @@ local function approximate_token_count(text) return math.ceil(string.len(text) / 4) end --- Preserve valid native IDs. Foreign-provider IDs with other characters need a --- stable translation, shared by calls/results and collision-checked against all --- IDs in the prompt. Character replacement alone merges distinct identities. -local function tool_id_mapper(messages) - local used, translated = {}, {} - local reserved = false - local function reserve_ids() - -- Normal native IDs need no extra history scan or hashing. Reserve all - -- IDs only if a foreign-provider ID actually needs translation. - if reserved then return end - reserved = true - local function reserve(id) - if type(id) == "string" then used[id] = id end - end - for _, msg in ipairs(messages) do - reserve(msg.function_call_id) - if msg.function_call then reserve(msg.function_call.id) end - if type(msg.content) == "table" then - for _, part in ipairs(msg.content) do - if part.type == "function_call" or part.type == "tool_use" then reserve(part.id) end - end - end - end - end - return function(id) - if type(id) ~= "string" or id == "" then error("Tool call ID is required") end - if id:match("^[a-zA-Z0-9_-]+$") then return id end - if translated[id] then return translated[id] end - reserve_ids() - local digest, err = hash.sha256(id) - if err then error(err) end - local base = "tool-" .. digest - local candidate, suffix = base, 0 - while used[candidate] and used[candidate] ~= id do - suffix = suffix + 1 - candidate = base .. "-" .. suffix - end - used[candidate], translated[id] = id, candidate - return candidate - end -end - local function last_cacheable_message_position(messages) for message_index = #messages, 1, -1 do local content = messages[message_index].content or {} for block_index = #content, 1, -1 do local block = content[block_index] if block and ( - (block.type == "text" and block.text and block.text ~= "") + (block.type == "text" and not prompt.is_blank(block.text)) or block.type == "tool_use" or block.type == "tool_result" or block.type == "image" or block.type == "document" ) then @@ -230,7 +192,8 @@ local STALE_THINKING_FIELDS: {[string]: {[string]: boolean}} = { } -- A thinking or redacted_thinking block as the API takes it back: every field passes --- through unchanged except the stale fields of its type; other blocks yield nil. +-- through unchanged except the stale fields of its type and the recorded response +-- position, which only orders the replay; other blocks yield nil. local function thinking_block_shape(block) local stale = STALE_THINKING_FIELDS[block.type] if not stale then @@ -238,7 +201,7 @@ local function thinking_block_shape(block) end local shaped = {} for key, value in pairs(block) do - if not stale[key] then + if not stale[key] and key ~= "position" then shaped[key] = value end end @@ -291,7 +254,9 @@ local function convert_image_content(content_part) } end end - return content_part + local copy = {} + for key, value in pairs(content_part) do copy[key] = value end + return copy end local function process_content_array(content) @@ -375,13 +340,22 @@ local function consolidate_messages(messages) return result end -local function ensure_content_exists(messages) - for i = 1, #messages - 1 do - if not messages[i].content or #messages[i].content == 0 then - messages[i].content = {{ type = "text", text = "intercepted" }} +-- An assistant turn with no visible text and no call renders as nothing, thinking +-- included: there is nothing in it to replay against. +local function without_blank_turns(messages) + local kept = {} + for _, msg in ipairs(messages) do + local visible = msg.role ~= "assistant" + for _, block in ipairs(msg.role == "assistant" and msg.content or {}) do + if block.type ~= "thinking" and block.type ~= "redacted_thinking" then + visible = true + end + end + if visible then + table.insert(kept, msg) end end - return messages + return kept end function mapper.map_messages(contract_messages) @@ -392,7 +366,27 @@ function mapper.map_messages(contract_messages) } end - local tool_id = tool_id_mapper(contract_messages) + contract_messages = prompt.without_blank_text(contract_messages) + local tool_id = tool_ids.mapper(contract_messages, { pattern = tool_ids.NATIVE_PATTERN }) + local order = thinking.new_turn_order() + local stale_thinking = {} + + local function add_thinking_blocks(content_blocks, msg) + if not (msg.metadata and msg.metadata.thinking_blocks) then + return + end + for _, thinking_block in ipairs(msg.metadata.thinking_blocks) do + if order.keeps_thinking(thinking_block) then + local wire_block = thinking_block_shape(thinking_block) or thinking_block + order.place_thinking(wire_block, thinking_block) + if msg.metadata.stale_thinking then + stale_thinking[wire_block] = true + end + table.insert(content_blocks, wire_block) + end + end + end + local claude_messages = {} local system_blocks = {} local system_cache_positions = {} @@ -400,6 +394,10 @@ function mapper.map_messages(contract_messages) local in_system_phase = true for i, msg in ipairs(contract_messages) do + if msg.role ~= prompt.ROLE.ASSISTANT and msg.role ~= prompt.ROLE.FUNCTION_CALL + and not prompt.transparent_to_assistant_turn(msg) then + order.leave_turn() + end if prompt.is_positional_note(msg) or msg.role == prompt.ROLE.DEVELOPER then -- Notes join the user turn they follow as blocks after that turn's -- content. Existing blocks are never edited, so the rendering does @@ -467,18 +465,17 @@ function mapper.map_messages(contract_messages) local arguments = normalize_tool_arguments(msg.function_call.arguments) local content_blocks = {} - if msg.metadata and msg.metadata.thinking_blocks then - for _, thinking_block in ipairs(msg.metadata.thinking_blocks) do - table.insert(content_blocks, thinking_block_shape(thinking_block) or thinking_block) - end - end + order.enter_row(msg) + add_thinking_blocks(content_blocks, msg) - table.insert(content_blocks, { + local tool_use = { type = "tool_use", id = tool_id(msg.function_call.id), name = msg.function_call.name, input = arguments - }) + } + order.place_call(tool_use, msg.function_call.id) + table.insert(content_blocks, tool_use) table.insert(claude_messages, { role = "assistant", @@ -488,15 +485,20 @@ function mapper.map_messages(contract_messages) in_system_phase = false local content_blocks = {} - if msg.metadata and msg.metadata.thinking_blocks then - for _, thinking_block in ipairs(msg.metadata.thinking_blocks) do - table.insert(content_blocks, thinking_block_shape(thinking_block) or thinking_block) - end - end + local layout = order.enter_row(msg) + add_thinking_blocks(content_blocks, msg) local regular_content = process_content_array(msg.content) - if type(regular_content) == "string" and regular_content ~= "" then + if layout then + for _, entry in ipairs(layout.texts) do + local text_block = { type = "text", text = entry.text } + order.place_text(text_block, entry.position) + table.insert(content_blocks, text_block) + end + end + + if type(regular_content) == "string" and not prompt.is_blank(regular_content) and not layout then table.insert(content_blocks, { type = "text", text = regular_content @@ -505,13 +507,15 @@ function mapper.map_messages(contract_messages) for _, part in ipairs(regular_content) do if part.type == "function_call" then local arguments = normalize_tool_arguments(part.arguments) - table.insert(content_blocks, { + local tool_use = { type = "tool_use", id = tool_id(part.id), name = part.name, input = arguments - }) - elseif part.type == "text" and part.text and part.text ~= "" then + } + order.place_call(tool_use, part.id) + table.insert(content_blocks, tool_use) + elseif part.type == "text" and not prompt.is_blank(part.text) and not layout then table.insert(content_blocks, part) elseif part.type ~= "text" then table.insert(content_blocks, part) @@ -562,11 +566,17 @@ function mapper.map_messages(contract_messages) end claude_messages = consolidate_messages(claude_messages) - claude_messages = ensure_content_exists(claude_messages) + for _, msg in ipairs(claude_messages) do + if msg.role == "assistant" then + msg.content = order.rebuild(msg.content) + end + end + claude_messages = without_blank_turns(claude_messages) return { messages = claude_messages, - system = #system_blocks > 0 and system_blocks or nil + system = #system_blocks > 0 and system_blocks or nil, + stale_thinking = next(stale_thinking) and stale_thinking or nil } end @@ -671,8 +681,7 @@ function mapper.map_options(contract_options, accepts): (table, table, string?) claude_options.output_config = { effort = thinking.effort_level(contract_options.thinking_effort) } end elseif mode == "budget" and contract_options.thinking_effort and contract_options.thinking_effort > 0 then - local thinking_budget = 1024 + (24000 - 1024) * (contract_options.thinking_effort / 100) - thinking_budget = math.floor(thinking_budget + 0.5) + local thinking_budget, max_tokens = route.budget_output(contract_options) claude_options.thinking = { type = "enabled", @@ -686,9 +695,7 @@ function mapper.map_options(contract_options, accepts): (table, table, string?) claude_options.temperature = nil end - if not claude_options.max_tokens or claude_options.max_tokens <= thinking_budget then - claude_options.max_tokens = thinking_budget + 1024 - end + claude_options.max_tokens = max_tokens end return claude_options, adjusted @@ -708,8 +715,13 @@ local function replays_thinking(messages): boolean end -- Binds the thinking blocks a Messages payload replays to drop_block and returns the --- request betas its block_binding requires, or nil when the payload carries none. -function mapper.bind_thinking(payload): {string}? +-- request betas its block_binding requires, or nil when the payload carries none. A +-- thinking object that cannot carry a binding sends none of the blocks in +-- `stale_blocks` (from map_messages): unbound, the API would reject them. +function mapper.bind_thinking(payload, stale_blocks): {string}? + if not thinking.is_bindable(payload.thinking) then + thinking.remove_blocks(payload.messages, stale_blocks) + end if replays_thinking(payload.messages) then thinking.bind_replayed_blocks(payload.thinking) end @@ -723,6 +735,7 @@ function mapper.extract_response_content(claude_response) local content_text = "" local tool_calls = {} local thinking_blocks = {} + local entries = {} if not claude_response or not claude_response.content then return { @@ -735,21 +748,26 @@ function mapper.extract_response_content(claude_response) for _, block in ipairs(claude_response.content) do if block.type == "text" then content_text = content_text .. (block.text or "") + table.insert(entries, { kind = "text", text = block.text }) elseif block.type == "tool_use" then + table.insert(entries, { kind = "tool_use", id = block.id }) table.insert(tool_calls, { id = block.id, name = block.name or "", arguments = block.input or {} }) elseif block.type == "thinking" or block.type == "redacted_thinking" then - table.insert(thinking_blocks, thinking_block_shape(block)) + local shaped = thinking_block_shape(block) + table.insert(entries, { kind = "thinking", block = shaped }) + table.insert(thinking_blocks, shaped) end end return { content = content_text, tool_calls = tool_calls, - thinking_blocks = thinking_blocks + thinking_blocks = thinking_blocks, + content_positions = thinking.record_positions(entries) } end @@ -793,6 +811,7 @@ function mapper.format_success_response(claude_response, model, name_to_id_map) result.metadata.thinking = thinking_content result.metadata.thinking_blocks = extracted.thinking_blocks + result.metadata.content_positions = extracted.content_positions return result end @@ -822,6 +841,7 @@ function mapper.format_streaming_response(client_result, name_to_id_map, usage, result.metadata.thinking = thinking_content result.metadata.thinking_blocks = thinking_blocks + result.metadata.content_positions = client_result.content_positions return result end diff --git a/src/llm/src/claude/structured_output.lua b/src/llm/src/claude/structured_output.lua index 8c60461..9e9f021 100644 --- a/src/llm/src/claude/structured_output.lua +++ b/src/llm/src/claude/structured_output.lua @@ -174,6 +174,9 @@ function structured_output_handler.handler(contract_args) end local mapped_messages = structured_output_handler._mapper.map_messages(contract_args.messages) + if #mapped_messages.messages == 0 then + return nil, err:kind(output.ERROR_TYPE.INVALID_REQUEST):message(output.NO_TURN_MESSAGE):build() + end local mapped_options, adjusted, options_err = structured_output_handler._mapper.map_options(contract_args.options or {}, contract_args.accepts) if options_err then return nil, err:kind(output.ERROR_TYPE.INVALID_REQUEST):message(options_err):build() @@ -190,7 +193,7 @@ function structured_output_handler.handler(contract_args) local native_payload = { model = contract_args.model, messages = mapped_messages.messages, - max_tokens = mapped_options.max_tokens or 2000 + max_tokens = mapped_options.max_tokens or mapper.DEFAULT_MAX_TOKENS } if mapped_messages.system then native_payload.system = mapped_messages.system @@ -208,7 +211,7 @@ function structured_output_handler.handler(contract_args) end -- The binding depends on the thinking object as sent, so it is the last change to the payload. - local native_betas = structured_output_handler._mapper.bind_thinking(native_payload) + local native_betas = structured_output_handler._mapper.bind_thinking(native_payload, mapped_messages.stale_thinking) local response, request_err = structured_output_handler._client.request( structured_output_handler._client.ENDPOINTS.MESSAGES, native_payload, @@ -259,7 +262,7 @@ function structured_output_handler.handler(contract_args) messages = mapped_messages.messages, tools = claude_tools, tool_choice = tool_choice, - max_tokens = mapped_options.max_tokens or 2000 + max_tokens = mapped_options.max_tokens or mapper.DEFAULT_MAX_TOKENS } if mapped_messages.system then @@ -283,7 +286,7 @@ function structured_output_handler.handler(contract_args) } -- The binding depends on the thinking object as sent, so it is the last change to the payload. - request_options.beta_features = structured_output_handler._mapper.bind_thinking(claude_payload) + request_options.beta_features = structured_output_handler._mapper.bind_thinking(claude_payload, mapped_messages.stale_thinking) local response, request_err = structured_output_handler._client.request( structured_output_handler._client.ENDPOINTS.MESSAGES, diff --git a/src/llm/src/discovery/providers.lua b/src/llm/src/discovery/providers.lua index 3e8c797..0f113e5 100644 --- a/src/llm/src/discovery/providers.lua +++ b/src/llm/src/discovery/providers.lua @@ -109,11 +109,9 @@ function providers.open(provider_id, context_overrides) return instance end --- Whether the provider's driver binding declares that it owns the legacy --- reasoning_model_request meaning (contract.binding meta.legacy_reasoning_flag). --- Resolves the provider entry's data.driver.id the same way providers.open does, --- so a custom provider entry bound to a framework driver is covered too. -function providers.driver_declares_legacy_reasoning_flag(provider_id): (boolean?, string | error | nil) +-- The contract.binding entry behind a provider entry's data.driver.id, resolved the +-- way providers.open does; nil for a provider entry without a driver. +local function driver_binding(provider_id): (any?, string | error | nil) local provider_entry, err = providers._registry.get(provider_id) if err then return nil, error_context.prefix(err, "Registry error: ") @@ -124,7 +122,7 @@ function providers.driver_declares_legacy_reasoning_flag(provider_id): (boolean? local driver = provider_entry.data and provider_entry.data.driver if not driver or not driver.id then - return false, nil + return nil, nil end local binding_entry, binding_err = providers._registry.get(tostring(driver.id)) @@ -134,8 +132,31 @@ function providers.driver_declares_legacy_reasoning_flag(provider_id): (boolean? if not binding_entry then return nil, "Driver binding not found: " .. tostring(driver.id) end + return binding_entry, nil +end + + +-- Whether the provider's driver binding declares that it owns the legacy +-- reasoning_model_request meaning (contract.binding meta.legacy_reasoning_flag). +-- Resolves the provider entry's data.driver.id the same way providers.open does, +-- so a custom provider entry bound to a framework driver is covered too. +function providers.driver_declares_legacy_reasoning_flag(provider_id): (boolean?, string | error | nil) + local binding, err = driver_binding(provider_id) + if err then + return nil, err + end + return binding ~= nil and binding.meta ~= nil and binding.meta.legacy_reasoning_flag == true, nil +end - return binding_entry.meta ~= nil and binding_entry.meta.legacy_reasoning_flag == true, nil +-- What a driver sends when a request leaves the output unset +-- (contract.binding meta.output_policy): { thinking, default_max_tokens }. +-- A driver that declares none lets the provider decide, so nothing can be reserved. +function providers.driver_output_policy(provider_id): (table?, string | error | nil) + local binding, err = driver_binding(provider_id) + if err then + return nil, err + end + return binding and binding.meta and binding.meta.output_policy or nil, nil end return providers diff --git a/src/llm/src/google/generate.lua b/src/llm/src/google/generate.lua index b79f42a..6e1c1ee 100644 --- a/src/llm/src/google/generate.lua +++ b/src/llm/src/google/generate.lua @@ -30,6 +30,10 @@ function generate.handler(contract_args) model = contract_args.model }) + if #messages == 0 then + return nil, err:kind(output.ERROR_TYPE.INVALID_REQUEST):message(output.NO_TURN_MESSAGE):build() + end + local payload = { contents = messages, generationConfig = {} diff --git a/src/llm/src/google/mapper.lua b/src/llm/src/google/mapper.lua index 4677b34..dab9d7e 100644 --- a/src/llm/src/google/mapper.lua +++ b/src/llm/src/google/mapper.lua @@ -145,6 +145,7 @@ end function mapper.map_messages(contract_messages, options) options = options or {} + contract_messages = prompt.without_blank_text(contract_messages) local processed_messages = {} local system_instructions = {} local provider_call_ids = {} @@ -180,7 +181,7 @@ function mapper.map_messages(contract_messages, options) i = i + 1 elseif msg.role == "assistant" then local assistant_msg = mapper.standardize_content(msg.content) - if assistant_msg ~= "" then + if not prompt.is_blank(assistant_msg) then table.insert(processed_messages, { role = "model", parts = {{ text = assistant_msg }} }) end diff --git a/src/llm/src/google/structured_output.lua b/src/llm/src/google/structured_output.lua index 94ba475..b375f82 100644 --- a/src/llm/src/google/structured_output.lua +++ b/src/llm/src/google/structured_output.lua @@ -75,6 +75,10 @@ function structured_output.handler(contract_args) model = contract_args.model }) + if #messages == 0 then + return nil, err:kind(output.ERROR_TYPE.INVALID_REQUEST):message(output.NO_TURN_MESSAGE):build() + end + local payload = { contents = messages, generationConfig = { diff --git a/src/llm/src/llm.lua b/src/llm/src/llm.lua index 9840deb..249784f 100644 --- a/src/llm/src/llm.lua +++ b/src/llm/src/llm.lua @@ -593,11 +593,25 @@ local function prepare_route(contract_args, provider_info: any, options, provide remove("thinking_effort") end - local limit = output_limit(card) + local policy, policy_err = providers_module.driver_output_policy(provider_info.id) + if policy_err then + return nil, native_error(policy_err, errors.INTERNAL, false) + end local requested = tonumber(contract_args.options.max_tokens) - if limit ~= nil and requested ~= nil and requested > limit then - adjusted.max_tokens = { requested = requested, sent = limit } - contract_args.options.max_tokens = limit + local plan = route.plan_output(requested, contract_args.options.thinking_effort, + policy and (accepts.thinking or policy.thinking), policy and policy.default_max_tokens, output_limit(card)) + if plan.max_tokens ~= requested then + contract_args.options.max_tokens = plan.max_tokens + if requested ~= nil then + adjusted.max_tokens = { requested = requested, sent = plan.max_tokens } + end + end + if plan.dropped_thinking then + remove("thinking_effort") + end + contract_args.options.thinking_budget_tokens = plan.budget + if plan.budget ~= plan.wanted_budget then + adjusted.thinking_budget = { requested = plan.wanted_budget, sent = plan.budget } end local strict_err = route.strict_error(tostring(provider_info.id), strict, adjusted) diff --git a/src/llm/src/openai/_index.yaml b/src/llm/src/openai/_index.yaml index f23a4a9..caaa525 100644 --- a/src/llm/src/openai/_index.yaml +++ b/src/llm/src/openai/_index.yaml @@ -157,6 +157,7 @@ entries: output: wippy.llm:output route: wippy.llm.util:route prompt: wippy.llm:prompt + tool_id: wippy.llm.util:tool_id # wippy.llm.openai:organization - name: organization diff --git a/src/llm/src/openai/generate.lua b/src/llm/src/openai/generate.lua index 4df0654..b829517 100644 --- a/src/llm/src/openai/generate.lua +++ b/src/llm/src/openai/generate.lua @@ -129,6 +129,10 @@ function generate_handler.handler(contract_args) model = contract_args.model }) + if #input_items == 0 then + return nil, err:kind(output.ERROR_TYPE.INVALID_REQUEST):message(output.NO_TURN_MESSAGE):build() + end + local payload = { model = contract_args.model, input = input_items diff --git a/src/llm/src/openai/mapper.lua b/src/llm/src/openai/mapper.lua index 27da449..a4942ff 100644 --- a/src/llm/src/openai/mapper.lua +++ b/src/llm/src/openai/mapper.lua @@ -1,4 +1,5 @@ local json = require("json") +local tool_ids = require("tool_id") local output = require("output") local route = require("route") local prompt = require("prompt") @@ -102,20 +103,24 @@ local function to_output_content_part(part) return { type = "output_text", text = tostring(part.text or "") } end +-- Input parts of a message, or nil when none has anything to send. local function normalize_input_content(content) - if content == nil or content == "" then - return { { type = "input_text", text = "" } } + if prompt.is_blank(content) then + return nil end if type(content) == "string" then return { { type = "input_text", text = content } } end if type(content) == "table" then local parts = {} - for i, p in ipairs(content) do - parts[i] = to_input_content_part(p) + for _, p in ipairs(content) do + local part = to_input_content_part(p) + if part.type ~= "input_text" or not prompt.is_blank(part.text) then + table.insert(parts, part) + end end if #parts == 0 then - return { { type = "input_text", text = "" } } + return nil end return parts end @@ -123,7 +128,7 @@ local function normalize_input_content(content) end local function normalize_output_content(content) - if content == nil or content == "" then + if prompt.is_blank(content) then return nil end if type(content) == "string" then @@ -131,8 +136,11 @@ local function normalize_output_content(content) end if type(content) == "table" then local parts = {} - for i, p in ipairs(content) do - parts[i] = to_output_content_part(p) + for _, p in ipairs(content) do + local part = to_output_content_part(p) + if not prompt.is_blank(part.text) then + table.insert(parts, part) + end end if #parts == 0 then return nil end return parts @@ -175,6 +183,7 @@ end function openai_mapper.extract_instructions(contract_messages) if not contract_messages then return nil, {} end + contract_messages = prompt.without_blank_text(contract_messages) local parts = {} local consumed = {} for i, msg in ipairs(contract_messages) do @@ -191,23 +200,31 @@ function openai_mapper.extract_instructions(contract_messages) return table.concat(parts, "\n\n"), consumed end +-- Responses API call_id: longer ids fail with string_above_max_length. +local MAX_CALL_ID_LENGTH = 64 + -- Map contract messages into the Responses API input[] array. -- Non-positional instructions are pulled out via extract_instructions(). function openai_mapper.map_messages(contract_messages, options) options = options or {} if not contract_messages then return {} end + contract_messages = prompt.without_blank_text(contract_messages) local _, consumed = openai_mapper.extract_instructions(contract_messages) + local wire_call_id = tool_ids.mapper(contract_messages, { max_length = MAX_CALL_ID_LENGTH }) local items = {} for i, msg in ipairs(contract_messages) do if not consumed[i] and msg.role ~= "cache_marker" then if msg.role == "user" or msg.role == "system" or msg.role == "developer" then - table.insert(items, { - type = "message", - role = msg.role, - content = normalize_input_content(msg.content) - }) + local content = normalize_input_content(msg.content) + if content then + table.insert(items, { + type = "message", + role = msg.role, + content = content + }) + end elseif msg.role == "assistant" then local content = normalize_output_content(msg.content) if content then @@ -221,7 +238,7 @@ function openai_mapper.map_messages(contract_messages, options) if msg.function_call and msg.function_call.id then table.insert(items, { type = "function_call", - call_id = msg.function_call.id, + call_id = wire_call_id(msg.function_call.id), name = msg.function_call.name, arguments = encode_arguments(msg.function_call.arguments) }) @@ -243,7 +260,7 @@ function openai_mapper.map_messages(contract_messages, options) end table.insert(items, { type = "function_call_output", - call_id = call_id, + call_id = wire_call_id(call_id), output = msg.is_error == true and json.encode({ error = out_text }) or out_text }) end diff --git a/src/llm/src/openai/structured_output.lua b/src/llm/src/openai/structured_output.lua index 4b4050e..2ef05a5 100644 --- a/src/llm/src/openai/structured_output.lua +++ b/src/llm/src/openai/structured_output.lua @@ -107,6 +107,10 @@ function structured_output_handler.handler(contract_args) model = contract_args.model }) + if #input_items == 0 then + return nil, err:kind(output.ERROR_TYPE.INVALID_REQUEST):message(output.NO_TURN_MESSAGE):build() + end + local payload = { model = contract_args.model, input = input_items, diff --git a/src/llm/src/openai_compat/_index.yaml b/src/llm/src/openai_compat/_index.yaml index 7e48b92..5c4fd1a 100644 --- a/src/llm/src/openai_compat/_index.yaml +++ b/src/llm/src/openai_compat/_index.yaml @@ -124,6 +124,7 @@ entries: - json imports: output: wippy.llm:output + prompt: wippy.llm:prompt route: wippy.llm.util:route # wippy.llm.openai_compat:organization @@ -189,6 +190,7 @@ entries: openai_client: wippy.llm.openai_compat:client openai_mapper: wippy.llm.openai_compat:mapper output: wippy.llm:output + prompt: wippy.llm:prompt wire_request: wippy.llm.util:wire_request method: handler pool: diff --git a/src/llm/src/openai_compat/mapper.lua b/src/llm/src/openai_compat/mapper.lua index 391fd0b..75c745a 100644 --- a/src/llm/src/openai_compat/mapper.lua +++ b/src/llm/src/openai_compat/mapper.lua @@ -1,6 +1,7 @@ local json = require("json") local output = require("output") local route = require("route") +local prompt = require("prompt") local openai_mapper = {} local MAX_CACHE_BREAKPOINTS = 4 @@ -156,18 +157,34 @@ end -- MESSAGE MAPPING FUNCTIONS +local function map_text_message(msg) + local content = msg.content + if msg.role == "developer" then + content = "" + if type(msg.content) == "string" then + content = msg.content + elseif type(msg.content) == "table" then + for _, part in ipairs(msg.content) do + if part.type == "text" then content = content .. part.text end + end + end + end + return { + role = msg.role == "developer" and "system" or msg.role, + content = normalize_content(content) + } +end + function openai_mapper.map_messages(contract_messages, options) options = options or {} - local processed_messages = {} + contract_messages = prompt.without_blank_text(contract_messages) + local processed_messages: {any} = {} local i = 1 local is_claude = is_claude_model(options.model) local cache_positions = {} while i <= #contract_messages do local msg = contract_messages[i] - -- Clear metadata - msg.metadata = nil - if msg.role == "cache_marker" then -- A marker after a string tool result cannot be represented as a -- content-part breakpoint without changing the provider payload. @@ -180,13 +197,8 @@ function openai_mapper.map_messages(contract_messages, options) end -- Skip cache marker message (don't add to processed_messages) i = i + 1 - elseif msg.role == "user" or msg.role == "system" then - -- Standard user/system messages - use structured content - local processed_msg = { - role = msg.role, - content = normalize_content(msg.content) - } - table.insert(processed_messages, processed_msg) + elseif msg.role == "user" or msg.role == "system" or msg.role == "developer" then + table.insert(processed_messages, map_text_message(msg)) i = i + 1 elseif msg.role == "assistant" then -- Assistant messages - ALWAYS use simple string content for OpenRouter compatibility @@ -207,6 +219,9 @@ function openai_mapper.map_messages(contract_messages, options) -- The prompt builder interleaves them: call_A, result_A, call_B, result_B local function_calls_found = {} local function_results_found = {} + local deferred_messages = {} + local open_calls = 0 + local answered = {} local scan_end = i - 1 local j = i while j <= #contract_messages do @@ -214,13 +229,33 @@ function openai_mapper.map_messages(contract_messages, options) if current_msg.role == "function_call" and current_msg.function_call and current_msg.function_call.id then table.insert(function_calls_found, current_msg) + if not answered[current_msg.function_call.id] then + open_calls = open_calls + 1 + end scan_end = j elseif current_msg.role == "function_result" then table.insert(function_results_found, current_msg) + if current_msg.function_call_id and not answered[current_msg.function_call_id] then + answered[current_msg.function_call_id] = true + for _, found in ipairs(function_calls_found) do + if found.function_call.id == current_msg.function_call_id then + open_calls = open_calls - 1 + break + end + end + end scan_end = j elseif current_msg.role == "assistant" or current_msg.role == "user" then break + elseif current_msg.role ~= "function_call" and current_msg.role ~= "cache_marker" and open_calls == 0 then + -- With every call answered the round is complete: a later call is + -- the next round, and this message stays in place between them. + break + elseif current_msg.role == "system" or current_msg.role == "developer" then + table.insert(deferred_messages, map_text_message(current_msg)) + scan_end = j end + if open_calls == 0 and #deferred_messages > 0 then break end j = j + 1 end @@ -246,7 +281,14 @@ function openai_mapper.map_messages(contract_messages, options) assistant_msg.tool_calls = nil end - table.insert(processed_messages, assistant_msg) + -- A turn with no visible text and no call renders as nothing; a turn with + -- calls and no visible text sends the empty content its calls need. + if prompt.is_blank(assistant_msg.content) then + assistant_msg.content = "" + end + if assistant_msg.tool_calls or assistant_msg.content ~= "" then + table.insert(processed_messages, assistant_msg) + end -- Emit function results as tool messages in order for _, result_msg in ipairs(function_results_found) do @@ -275,6 +317,9 @@ function openai_mapper.map_messages(contract_messages, options) end table.insert(processed_messages, tool_msg) end + for _, deferred in ipairs(deferred_messages) do + table.insert(processed_messages, deferred) + end -- Advance past all scanned messages i = scan_end + 1 @@ -312,24 +357,6 @@ function openai_mapper.map_messages(contract_messages, options) table.insert(processed_messages, tool_msg) i = i + 1 - elseif msg.role == "developer" then - -- Convert developer messages to system messages - local content = "" - if type(msg.content) == "string" then - content = msg.content - elseif type(msg.content) == "table" then - for _, part in ipairs(msg.content) do - if part.type == "text" then - content = content .. part.text - end - end - end - - table.insert(processed_messages, { - role = "system", - content = normalize_content(content) - }) - i = i + 1 elseif msg.role == "function_call" then -- Standalone function_calls without preceding assistant message if not msg.function_call or not msg.function_call.id then diff --git a/src/llm/src/openai_compat/structured_output.lua b/src/llm/src/openai_compat/structured_output.lua index 7f09605..9f15a3c 100644 --- a/src/llm/src/openai_compat/structured_output.lua +++ b/src/llm/src/openai_compat/structured_output.lua @@ -1,3 +1,4 @@ +local prompt = require("prompt") local openai_client = require("openai_client") local openai_mapper = require("openai_mapper") local output = require("output") @@ -105,7 +106,7 @@ function structured_output_handler.handler(contract_args) local openai_payload = { model = contract_args.model, - messages = contract_args.messages, + messages = prompt.without_blank_text(contract_args.messages), response_format = { type = "json_schema", json_schema = { diff --git a/src/llm/src/output.lua b/src/llm/src/output.lua index 584ead3..4c1a97f 100644 --- a/src/llm/src/output.lua +++ b/src/llm/src/output.lua @@ -470,6 +470,9 @@ function output.stream_error_details(started: boolean, deferred: ErrorInfo?): { end -- Message injected into conversation when LLM output is truncated mid-tool-call +-- The driver error for a request whose mapped conversation holds nothing to answer. +output.NO_TURN_MESSAGE = "Request has no conversation turn: every message is instructions or blank" + output.TRUNCATION_MSG = "Your previous response was truncated because it exceeded the maximum token limit. The tool calls you attempted were incomplete and have been discarded. Please retry with a shorter response. Break your work into smaller steps if needed." -- Detect LLM responses cut off by the token limit that produced nothing diff --git a/src/llm/src/prompt.lua b/src/llm/src/prompt.lua index c0027bc..6d01df1 100644 --- a/src/llm/src/prompt.lua +++ b/src/llm/src/prompt.lua @@ -53,6 +53,7 @@ type PromptBuilder = { add_function_result: (self: any, name: string, content: any, function_call_id: string?, options: FunctionResultOptions?) -> PromptBuilder, add_cache_marker: (self: any, marker_id: string?) -> PromptBuilder, drop_stale_thinking: (self: any, is_current: (msg: Message) -> boolean) -> (PromptBuilder?, string?), + mark_stale_thinking: (self: any, is_current: (msg: Message) -> boolean) -> PromptBuilder, get_messages: (self: any) -> {Message}, build: (self: any) -> {messages: {Message}}, clone: (self: any) -> PromptBuilder, @@ -147,10 +148,87 @@ function prompt.developer_instruction(text: string): string return "" .. text .. "" end +-- Whether text has nothing visible in it. This one rule decides which text blocks +-- reach a provider and which assistant turns render at all. +function prompt.is_blank(text: any): boolean + return text == nil or (type(text) == "string" and not text:find("%S")) +end + +local BLANKABLE_ROLES: {[string]: boolean} = { + [prompt.ROLE.USER] = true, + [prompt.ROLE.SYSTEM] = true, + [prompt.ROLE.DEVELOPER] = true, +} + +local function is_blank_part(part: any): boolean + if type(part) == "string" then return prompt.is_blank(part) end + if type(part) ~= "table" then return false end + return (part.type == prompt.CONTENT_TYPE.TEXT or part.type == "input_text") and prompt.is_blank(part.text) +end + +-- User, system and developer messages without their blank text. A message left +-- with no content goes entirely; the caller's tables stay as they are. +function prompt.without_blank_text(messages: {any}): {any} + local kept: {any} = table.create(#messages, 0) + for _, msg in ipairs(messages) do + local content = msg.content + if not BLANKABLE_ROLES[msg.role] then + table.insert(kept, msg) + elseif type(content) == "table" then + local parts: {any} = {} + for _, part in ipairs(content) do + if not is_blank_part(part) then table.insert(parts, part) end + end + if #parts == #content and #content > 0 then + table.insert(kept, msg) + elseif #parts > 0 then + local copy: any = {} + for key, value in pairs(msg) do copy[key] = value end + copy.content = parts + table.insert(kept, copy) + end + elseif not prompt.is_blank(content) then + table.insert(kept, msg) + end + end + return kept +end + --------------------------- -- Helper Functions --------------------------- +local function in_assistant_turn(msg: Message): boolean + return msg.role == prompt.ROLE.ASSISTANT or msg.role == prompt.ROLE.FUNCTION_CALL +end + +function prompt.transparent_to_assistant_turn(msg: Message): boolean + return msg.role == prompt.ROLE.CACHE_MARKER + or (msg.role == prompt.ROLE.SYSTEM and not prompt.is_positional_note(msg)) +end + +local function carries_thinking(msg: Message): boolean + if not in_assistant_turn(msg) or type(msg.metadata) ~= "table" then return false end + local blocks = msg.metadata.thinking_blocks + return type(blocks) == "table" and #blocks > 0 +end + +-- A copy of a message with a rewritten metadata table, so message tables the +-- caller shares with the builder keep theirs. +local function with_metadata(msg: Message, rewrite: (metadata: table) -> ()): Message + local copy: Message = {} :: Message + for key, value in pairs(msg :: table) do + (copy :: any)[key] = value + end + local metadata = {} + for key, value in pairs(msg.metadata :: table) do + metadata[key] = value + end + rewrite(metadata) + copy.metadata = metadata + return copy +end + -- Process function result content to extract images and clean text local function process_function_result_content(content: any): (any, {ContentPart}?) local parsed @@ -393,26 +471,16 @@ function prompt.new(messages: {Message}?) end end - local function in_turn(msg: Message): boolean - return msg.role == prompt.ROLE.ASSISTANT or msg.role == prompt.ROLE.FUNCTION_CALL - end - - local function carries(msg: Message): boolean - if not in_turn(msg) or type(msg.metadata) ~= "table" then return false end - local blocks = msg.metadata.thinking_blocks - return type(blocks) == "table" and #blocks > 0 - end - -- First unanswered call of each assistant turn, keyed by message index. local open_call: {[integer]: string} = {} local turn_start: integer? = nil for index, msg in ipairs(messages) do - if in_turn(msg) then + if in_assistant_turn(msg) then turn_start = turn_start or index - else + elseif not prompt.transparent_to_assistant_turn(msg) then turn_start = nil end - if turn_start then + if turn_start and in_assistant_turn(msg) then local ids: {any} = {} if msg.role == prompt.ROLE.FUNCTION_CALL then table.insert(ids, { @@ -420,9 +488,11 @@ function prompt.new(messages: {Message}?) name = msg.function_call and msg.function_call.name, }) else - for _, part in ipairs(msg.content or {}) do - if (part :: any).type == "function_call" then - table.insert(ids, { id = (part :: any).id or false, name = (part :: any).name }) + if type(msg.content) == "table" then + for _, part in ipairs(msg.content) do + if (part :: any).type == "function_call" then + table.insert(ids, { id = (part :: any).id or false, name = (part :: any).name }) + end end end end @@ -447,7 +517,7 @@ function prompt.new(messages: {Message}?) local current_round = #messages + 1 local seen_result, seen_user = false, false local index = #messages - while index >= 1 and not in_turn(messages[index]) do + while index >= 1 and not in_assistant_turn(messages[index]) do local role = messages[index].role if role == prompt.ROLE.FUNCTION_RESULT then seen_result = true end if role == prompt.ROLE.USER or role == prompt.ROLE.DEVELOPER @@ -457,8 +527,8 @@ function prompt.new(messages: {Message}?) index = index - 1 end if index >= 1 and not (seen_user and not seen_result) then - while index >= 1 and in_turn(messages[index]) do - current_round = index + while index >= 1 and (in_assistant_turn(messages[index]) or prompt.transparent_to_assistant_turn(messages[index])) do + if in_assistant_turn(messages[index]) then current_round = index end index = index - 1 end end @@ -466,7 +536,7 @@ function prompt.new(messages: {Message}?) local last_stale: integer? = nil for index = 1, current_round - 1 do local msg = messages[index] - if carries(msg) and not is_current(msg) then + if carries_thinking(msg) and not is_current(msg) then last_stale = index end end @@ -478,12 +548,12 @@ function prompt.new(messages: {Message}?) turn_start = nil for index = 1, last_stale do local msg = messages[index] - if in_turn(msg) then + if in_assistant_turn(msg) then turn_start = turn_start or index - else + elseif not prompt.transparent_to_assistant_turn(msg) then turn_start = nil end - if carries(msg) then + if carries_thinking(msg) then local open = turn_start and open_call[turn_start] if open then return nil, string.format( @@ -493,21 +563,38 @@ function prompt.new(messages: {Message}?) end end - -- Stripped messages are replaced by copies, so message tables the - -- caller shares with the builder keep their blocks. for _, index in ipairs(stale) do - local copy: Message = {} :: Message - for key, value in pairs(messages[index] :: table) do - (copy :: any)[key] = value + messages[index] = with_metadata(messages[index], function(metadata) + metadata.thinking_blocks = nil + end) + end + return self, nil + end + + -- Marks the blocks that stay in the request although they are stale, the ones + -- drop_stale_thinking keeps for the current round. Only the caller knows they + -- are stale and only the driver knows whether the thinking object it sends can + -- carry a binding, so the driver removes marked blocks when it cannot. Marks + -- follow the same front-only rule: every block up to the last stale one. + builder.mark_stale_thinking = function(self: any, is_current: (msg: Message) -> boolean) + local messages: {Message} = self.messages :: {Message} + local last_stale: integer? = nil + for index, msg in ipairs(messages) do + if carries_thinking(msg) and not is_current(msg) then + last_stale = index end - local metadata = {} - for key, value in pairs(messages[index].metadata :: table) do - if key ~= "thinking_blocks" then metadata[key] = value end + end + if not last_stale then + return self + end + for index = 1, last_stale do + if carries_thinking(messages[index]) then + messages[index] = with_metadata(messages[index], function(metadata) + metadata.stale_thinking = true + end) end - copy.metadata = metadata - messages[index] = copy end - return self, nil + return self end -- Get all messages in the current builder (with image processing) diff --git a/src/llm/src/util/_index.yaml b/src/llm/src/util/_index.yaml index 8b1b1ef..29ec6bf 100644 --- a/src/llm/src/util/_index.yaml +++ b/src/llm/src/util/_index.yaml @@ -50,9 +50,20 @@ entries: - fallback source: file://fallback.lua + - name: tool_id + kind: library.lua + meta: + type: library + comment: Stable translation of foreign tool call ids into ids a provider accepts + source: file://tool_id.lua + modules: + - hash + - name: thinking kind: library.lua source: file://thinking.lua + imports: + prompt: wippy.llm:prompt # wippy.llm.util:wire_request - name: wire_request diff --git a/src/llm/src/util/route.lua b/src/llm/src/util/route.lua index 008349b..6209527 100644 --- a/src/llm/src/util/route.lua +++ b/src/llm/src/util/route.lua @@ -105,6 +105,56 @@ function route.clean_options(options: table): table return clean end +-- Budget thinking takes a share of the response: the budget grows with the effort +-- and the response keeps room for an answer behind it. +local MIN_THINKING_BUDGET = 1024 +local MAX_THINKING_BUDGET = 24000 +local ANSWER_ROOM = 1024 + +function route.thinking_budget(effort: number): number + return math.floor(MIN_THINKING_BUDGET + (MAX_THINKING_BUDGET - MIN_THINKING_BUDGET) * (effort / 100) + 0.5) +end + +-- The output a request sends, computed once per route before the card limit and +-- admission: `mode` is the effective thinking mode, `default_max_tokens` what the +-- driver sends for an unset max_tokens and `limit` the card's output limit. +-- budget is nil when the request has no budget thinking; dropped_thinking says the +-- limit leaves no room for the smallest budget. +function route.plan_output(max_tokens: number?, effort: number?, mode: string?, default_max_tokens: number?, limit: number?): table + local thinks = mode == "budget" and effort ~= nil and effort > 0 + local wanted_budget = thinks and route.thinking_budget(effort :: number) or nil + local output = max_tokens or default_max_tokens + if wanted_budget and (not max_tokens or max_tokens <= wanted_budget) then + output = wanted_budget + ANSWER_ROOM + end + local capped = false + if limit and output and output > limit then + output = limit + capped = true + end + if not wanted_budget then + return { max_tokens = output } + end + if output <= MIN_THINKING_BUDGET then + return { max_tokens = math.min(max_tokens or default_max_tokens or output, output), dropped_thinking = true } + end + local budget = wanted_budget + if capped then + budget = math.min(budget, math.max(output - ANSWER_ROOM, MIN_THINKING_BUDGET)) + end + return { max_tokens = output, budget = budget, wanted_budget = wanted_budget } +end + +-- The budget and output a budget-thinking request sends. llm plans them before the +-- request; a driver called directly with no plan gets the same arithmetic. +function route.budget_output(options: table): (number, number) + if options.thinking_budget_tokens then + return options.thinking_budget_tokens, options.max_tokens + end + local plan = route.plan_output(options.max_tokens, options.thinking_effort, "budget", nil, nil) + return plan.budget, plan.max_tokens +end + -- A driver capability table: { name, defaults = {fact = value}, supported = {fact = {value = true}} }. -- `supported` only needs entries for facts a driver restricts beyond the global -- enum; a fact absent from `supported` accepts any globally valid value. diff --git a/src/llm/src/util/thinking.lua b/src/llm/src/util/thinking.lua index 08f62f3..bd654b1 100644 --- a/src/llm/src/util/thinking.lua +++ b/src/llm/src/util/thinking.lua @@ -1,3 +1,5 @@ +local prompt = require("prompt") + local thinking = {} -- Beta that admits thinking.block_binding and adds input_transformations to responses. @@ -17,11 +19,274 @@ function thinking.bind_replayed_blocks(config: {[string]: any}?) config.block_binding = { prefix_mismatch_behavior = "drop_block" } end +-- Whether a thinking object can carry block_binding. +function thinking.is_bindable(config: {[string]: any}?): boolean + return type(config) == "table" and BINDABLE_TYPES[config.type] == true +end + +-- Removes the wire blocks in `stale` from every message that holds one. +function thinking.remove_blocks(messages: {any}?, stale: {[any]: boolean}?) + if not messages or not stale or next(stale) == nil then + return + end + for _, message in ipairs(messages) do + if type(message.content) == "table" then + local kept = {} + for _, block in ipairs(message.content) do + if not stale[block] then table.insert(kept, block) end + end + if #kept ~= #message.content then + message.content = kept + end + end + end +end + -- Whether a thinking object carries block_binding, which the API admits only with BINDING_BETA. function thinking.is_bound(config: {[string]: any}?): boolean return type(config) == "table" and config.block_binding ~= nil end +-- Records where a response's blocks sit, given `entries` in response order: each +-- { kind = "thinking", block = }, { kind = "text", text = ... } or +-- { kind = "tool_use", id = ... }. A thinking block keeps its position on itself and +-- the others come back as the list for metadata.content_positions. A response that +-- is thinking, then at most one text, then calls replays in that order without any +-- record, so it gets none. +function thinking.record_positions(entries: {any}): {any}? + local present: {any} = {} + for _, entry in ipairs(entries) do + if entry.kind ~= "text" or (entry.text or "") ~= "" then + table.insert(present, entry) + end + end + + local stages: {[string]: number} = { thinking = 1, text = 2, tool_use = 3 } + local stage, texts = 1, 0 + local plain = true + for _, entry in ipairs(present) do + if stages[entry.kind] < stage then plain = false end + stage = stages[entry.kind] + if entry.kind == "text" then texts = texts + 1 end + end + if plain and texts <= 1 then + return nil + end + + local positions: {any} = {} + for position, entry in ipairs(present) do + if entry.kind == "thinking" then + entry.block.position = position + elseif entry.kind == "text" then + table.insert(positions, { type = "text", position = position, text = entry.text }) + else + table.insert(positions, { type = "tool_use", position = position, id = entry.id }) + end + end + return positions +end + +local function row_text(content: any): string + if type(content) == "string" then return content end + local text = "" + if type(content) == "table" then + for _, part in ipairs(content) do + if part.type == "text" and type(part.text) == "string" then + text = text .. part.text + end + end + end + return text +end + +local function without_outer_whitespace(text: string): string + return text:match("^%s*(.-)%s*$") :: string +end + +-- Thinking at or after a missing call no longer has its signed prefix. +local function thinking_limit(last: number, calls: {[any]: number}, retained: {[any]: boolean}): number + for id, position in pairs(calls) do + if not retained[id] then + last = math.min(last, position) + end + end + return last +end + +-- Takes the calls out of a response's metadata when the caller discards them, along +-- with the thinking that led to them. `text` is the response's text. +function thinking.discard_calls(metadata: any, text: any) + local blocks = metadata.thinking_blocks + local positions = metadata.content_positions + if type(positions) ~= "table" then + if prompt.is_blank(text) then + metadata.thinking_blocks = {} + end + return + end + + local limit = 0 + local calls = {} + local kept_positions = {} + for _, entry in ipairs(positions) do + if entry.type == "text" then + table.insert(kept_positions, entry) + if not prompt.is_blank(entry.text) then limit = math.max(limit, entry.position) end + elseif entry.type == "tool_use" then + calls[entry.id] = entry.position + end + end + limit = thinking_limit(limit, calls, {}) + local kept_blocks = {} + for _, block in ipairs(type(blocks) == "table" and blocks or {}) do + if block.position and block.position < limit then + table.insert(kept_blocks, block) + end + end + metadata.thinking_blocks = kept_blocks + metadata.content_positions = limit > 0 and kept_positions or nil +end + +-- A row's recorded layout, or nil when it has none or its text is no longer the +-- text the layout recorded: the row is what the host stands behind. Providers +-- compare neither whitespace-only text blocks nor the outer whitespace of a block, so +-- the layout holds only visible text, and whitespace a host stripped from the row's +-- ends is not an edit. +local function read_layout(msg: any): any? + local positions = type(msg.metadata) == "table" and msg.metadata.content_positions or nil + if type(positions) ~= "table" then + return nil + end + local layout = { texts = {}, calls = {} } + local text = "" + for _, entry in ipairs(positions) do + if entry.type == "text" then + if not prompt.is_blank(entry.text) then + table.insert(layout.texts, entry) + end + text = text .. entry.text + elseif entry.type == "tool_use" then + layout.calls[entry.id] = entry.position + end + end + if without_outer_whitespace(text) ~= without_outer_whitespace(row_text(msg.content)) then + return nil + end + return layout +end + +local function is_wire_thinking(block: any): boolean + return block.type == "thinking" or block.type == "redacted_thinking" or block.reasoningContent ~= nil +end + +-- Tracks the wire blocks of assistant rows against their recorded layouts and +-- rebuilds response order once the rows are consolidated into one turn. +function thinking.new_turn_order() + local placed: {[any]: any} = {} + local attached: {[any]: boolean} = {} + local active: any = nil + local order = {} + + -- Returns the row's own layout, whose text blocks replace the row's text. + function order.enter_row(msg: any): any? + local layout = read_layout(msg) + if type(msg.metadata) == "table" and msg.metadata.content_positions ~= nil then + active = layout + end + return layout + end + + function order.leave_turn() + active = nil + end + + function order.keeps_thinking(stored_block: any): boolean + if stored_block.type == "thinking" and (stored_block.signature or "") == "" then + return false + end + return true + end + + function order.place_thinking(wire_block: any, stored_block: any) + if active and stored_block.position then + placed[wire_block] = { layout = active, position = stored_block.position } + end + end + + function order.place_text(wire_block: any, position: number) + if active then + placed[wire_block] = { layout = active, position = position } + end + end + + function order.place_call(wire_block: any, id: any) + local position = active and active.calls[id] + if position then + placed[wire_block] = { layout = active, position = position, call_id = id } + end + end + + -- A block that belongs right behind `anchor`, wherever the anchor ends up. + function order.place_behind(wire_block: any, anchor: any) + attached[wire_block] = true + local place = placed[anchor] + if place then + placed[wire_block] = { layout = place.layout, position = place.position + 0.5 } + end + end + + -- The content with each run of blocks from one layout in response order. Blocks + -- with no recorded place keep theirs, and they end a run. + function order.rebuild(content: {any}): {any} + local limits: {[any]: number} = {} + local retained: {[any]: {[any]: boolean}} = {} + for _, block in ipairs(content) do + local place = placed[block] + if place and not is_wire_thinking(block) and not attached[block] then + limits[place.layout] = math.max(limits[place.layout] or 0, place.position) + retained[place.layout] = retained[place.layout] or {} + if place.call_id then retained[place.layout][place.call_id] = true end + end + end + for layout, last in pairs(limits) do + limits[layout] = thinking_limit(last, layout.calls, retained[layout] :: {[any]: boolean}) + end + local rebuilt: {any} = {} + local index = 1 + while index <= #content do + local place = placed[content[index]] + if not place then + table.insert(rebuilt, content[index]) + index = index + 1 + else + local run: {any} = {} + while content[index] and placed[content[index]] and placed[content[index]].layout == place.layout do + table.insert(run, content[index]) + index = index + 1 + end + table.sort(run, function(a, b) return placed[a].position < placed[b].position end) + for _, block in ipairs(run) do + if not is_wire_thinking(block) or placed[block].position < (limits[place.layout] or 0) then + table.insert(rebuilt, block) + end + end + end + end + -- Each emitted turn ends on visible content, even when persisted rows + -- split a response or carry thinking without recorded positions. + local last = #rebuilt + while last > 0 and (is_wire_thinking(rebuilt[last]) or attached[rebuilt[last]]) do + last = last - 1 + end + for index = #rebuilt, last + 1, -1 do + if is_wire_thinking(rebuilt[index]) then table.remove(rebuilt, index) end + end + return rebuilt + end + + return order +end + function thinking.effort_level(effort: number): string if effort >= 100 then return "max" end if effort >= 80 then return "xhigh" end diff --git a/src/llm/src/util/tool_id.lua b/src/llm/src/util/tool_id.lua new file mode 100644 index 0000000..b252955 --- /dev/null +++ b/src/llm/src/util/tool_id.lua @@ -0,0 +1,63 @@ +local hash = require("hash") + +local tool_id = {} + +-- The tool call ids Claude and Bedrock accept. A provider's id format can leave +-- another provider's ids outside it after a handoff. +tool_id.NATIVE_PATTERN = "^[a-zA-Z0-9_-]+$" + +local function fit(base: string, suffix: string, max_length: number?): string + if max_length and #base + #suffix > max_length then + base = base:sub(1, max_length - #suffix) + end + return base .. suffix +end + +-- A function mapping a conversation's tool call ids to ids a provider accepts, under +-- `limits`: { pattern?, max_length? }. Ids within the limits pass unchanged. Others +-- get a translation derived from a digest of the id, the same on every request because +-- it is part of the prefix, shared by calls and results, and collision-checked against +-- all ids in the conversation: character replacement or truncation alone would merge +-- distinct identities. +function tool_id.mapper(messages: {any}, limits: table): (id: any) -> string + local used, translated = {}, {} + local reserved = false + local function reserve_ids() + -- Normal native ids need no extra history scan or hashing. Reserve all + -- ids only if a foreign-provider id actually needs translation. + if reserved then return end + reserved = true + local function reserve(id) + if type(id) == "string" then used[id] = id end + end + for _, msg in ipairs(messages) do + reserve(msg.function_call_id) + if msg.function_call then reserve(msg.function_call.id) end + if type(msg.content) == "table" then + for _, part in ipairs(msg.content) do + if part.type == "function_call" or part.type == "tool_use" then reserve(part.id) end + end + end + end + end + return function(id) + if type(id) ~= "string" or id == "" then error("Tool call ID is required") end + local fits_pattern = not limits.pattern or id:match(limits.pattern) + local fits_length = not limits.max_length or #id <= limits.max_length + if fits_pattern and fits_length then return id end + if translated[id] then return translated[id] end + reserve_ids() + local digest, err = hash.sha256(id) + if err then error(err) end + local base = "tool-" .. digest + local candidate, suffix = fit(base, "", limits.max_length), 0 + while used[candidate] and used[candidate] ~= id do + suffix = suffix + 1 + candidate = fit(base, "-" .. suffix, limits.max_length) + end + used[candidate], translated[id] = id, candidate + return candidate + end +end + +return tool_id diff --git a/src/llm/test/_index.yaml b/src/llm/test/_index.yaml index bafc9a4..f456c0c 100644 --- a/src/llm/test/_index.yaml +++ b/src/llm/test/_index.yaml @@ -256,6 +256,40 @@ entries: test: wippy.test:test method: run_tests + # app:replay_safety_test + - name: replay_safety_test + kind: function.lua + meta: + name: Replay Safety Test + type: test + group: LLM / General + source: file://replay_safety_test.lua + modules: [json, registry] + imports: + llm: wippy.llm:llm + prompt: wippy.llm:prompt + providers: wippy.llm.discovery:providers + thinking: wippy.llm.util:thinking + claude_mapper: wippy.llm.claude:mapper + claude_client: wippy.llm.claude:client + claude_generate: wippy.llm.claude:generate + claude_structured: wippy.llm.claude:structured_output + bedrock_mapper: wippy.llm.bedrock:mapper + bedrock_client: wippy.llm.bedrock:client + bedrock_generate: wippy.llm.bedrock:generate + bedrock_structured: wippy.llm.bedrock:structured_output + openai_mapper: wippy.llm.openai:mapper + openai_generate: wippy.llm.openai:generate + openai_structured: wippy.llm.openai:structured_output + compat_mapper: wippy.llm.openai_compat:mapper + compat_generate: wippy.llm.openai_compat:generate + compat_structured: wippy.llm.openai_compat:structured_output + google_mapper: wippy.llm.google:mapper + google_generate: wippy.llm.google:generate + google_structured: wippy.llm.google:structured_output + test: wippy.test:test + method: run_tests + # app:output_test - name: output_test kind: function.lua diff --git a/src/llm/test/bedrock/mapper_test.lua b/src/llm/test/bedrock/mapper_test.lua index 0cf48e3..20e5d16 100644 --- a/src/llm/test/bedrock/mapper_test.lua +++ b/src/llm/test/bedrock/mapper_test.lua @@ -507,8 +507,7 @@ local function define_tests() }) test.not_nil(((result.messages[1].content :: any)[2] :: any).cachePoint) - test.not_nil(((result.messages[2].content :: any)[1] :: any).reasoningContent) - test.is_nil(result.messages[2].content[2]) + test.is_nil(result.messages[2]) end) it("should consolidate consecutive same-role messages", function() diff --git a/src/llm/test/claude/mapper_test.lua b/src/llm/test/claude/mapper_test.lua index 4adc343..969b62c 100644 --- a/src/llm/test/claude/mapper_test.lua +++ b/src/llm/test/claude/mapper_test.lua @@ -641,6 +641,24 @@ local function define_tests() test.eq(first_system.cache_control.type, "ephemeral") end) + it("marks copies of caller system, user and assistant text blocks", function() + local rows = { + { role = "system", content = { { type = "text", text = "Base" } } }, + { role = "cache_marker" }, + { role = "user", content = { { type = "text", text = "Question" } } }, + { role = "cache_marker" }, + { role = "assistant", content = { { type = "text", text = "Answer" } } }, + { role = "cache_marker" }, + } + local before = json.decode((json.encode(rows))) + local mapped = mapper.map_messages(rows) + + test.eq(mapped.system[1].cache_control.type, "ephemeral") + test.eq(mapped.messages[1].content[1].cache_control.type, "ephemeral") + test.eq(mapped.messages[2].content[1].cache_control.type, "ephemeral") + test.is_true(same(rows, before), "cache marking leaves caller blocks unchanged") + end) + it("should cache a tool-result history tail", function() local result = mapper.map_messages({ { role = "user", content = {{ type = "text", text = "Weather?" }} }, @@ -720,8 +738,7 @@ local function define_tests() }) test.not_nil((result.messages[1].content[1] :: any).cache_control) - test.eq(result.messages[2].content[1].type, "thinking") - test.is_nil(result.messages[2].content[1].cache_control) + test.is_nil(result.messages[2]) end) it("should keep the marked prefix stable when a dynamic developer note follows", function() diff --git a/src/llm/test/google/mapper_test.lua b/src/llm/test/google/mapper_test.lua index c80dd12..c4704ee 100644 --- a/src/llm/test/google/mapper_test.lua +++ b/src/llm/test/google/mapper_test.lua @@ -499,9 +499,7 @@ local function define_tests() local google_messages, _ = mapper.map_messages(contract_messages) - tests.eq(#google_messages, 1) - local empty_str_parts = google_messages[1].parts :: any - tests.eq(empty_str_parts[1].text, "") + tests.eq(#google_messages, 0) end) it("should skip assistant messages with empty content", function() diff --git a/src/llm/test/llm_test.lua b/src/llm/test/llm_test.lua index 77bd05c..0d37c87 100644 --- a/src/llm/test/llm_test.lua +++ b/src/llm/test/llm_test.lua @@ -232,6 +232,7 @@ local function define_tests() driver_declares_legacy_reasoning_flag = function(provider_id) return provider_id == "wippy.llm.openai:provider" end, + driver_output_policy = function() return nil, nil end, last_structured_output_args = nil, last_embed_args = nil, last_status_args = nil, diff --git a/src/llm/test/native_errors_test.lua b/src/llm/test/native_errors_test.lua index b03d55d..9c1518f 100644 --- a/src/llm/test/native_errors_test.lua +++ b/src/llm/test/native_errors_test.lua @@ -32,7 +32,7 @@ return test.run_cases(function() local cause = errors.new({ kind = kind, retryable = kind == errors.INTERNAL, message = "provider refused", details = { operation = method } }) llm._providers = { - driver_declares_legacy_reasoning_flag = function() return false end, + driver_declares_legacy_reasoning_flag = function() return false end, driver_output_policy = function() return nil, nil end, open = function() return { [method] = function() return nil, cause end } end, } local result, err = call(method, { model = "test-model", provider_id = direct and "provider" or nil, fallback = false }) @@ -94,7 +94,7 @@ return test.run_cases(function() it("retains the exact text of untyped " .. method .. " provider failures", function() llm._providers = { - driver_declares_legacy_reasoning_flag = function() return false end, + driver_declares_legacy_reasoning_flag = function() return false end, driver_output_policy = function() return nil, nil end, open = function() return { [method] = function() return nil, "provider refused" end } end, } local result, err = call(method, { model = "test-model" }) @@ -159,7 +159,7 @@ return test.run_cases(function() {name = "large", max_tokens = 2000, providers = {{id = "provider", provider_model = "large"}}}, } end} llm._providers = { - driver_declares_legacy_reasoning_flag = function() return false end, + driver_declares_legacy_reasoning_flag = function() return false end, driver_output_policy = function() return nil, nil end, open = function() return {generate = function(_, args) if args.model == "small" then return nil, errors.new({kind = errors.INVALID, message = "context too large", @@ -303,7 +303,7 @@ return test.run_cases(function() } } end } llm._providers = { - driver_declares_legacy_reasoning_flag = function() return false end, + driver_declares_legacy_reasoning_flag = function() return false end, driver_output_policy = function() return nil, nil end, open = function(id) return { generate = function() return nil, errors.new({ kind = errors.RATE_LIMITED, retryable = true, diff --git a/src/llm/test/prompt_test.lua b/src/llm/test/prompt_test.lua index 2d8fe6b..84938aa 100644 --- a/src/llm/test/prompt_test.lua +++ b/src/llm/test/prompt_test.lua @@ -190,6 +190,54 @@ local function define_tests() return builder end + for _, role in ipairs({ "cache_marker", "system" }) do + local function transparent_row() + return { role = role, content = { prompt.text("policy") }, marker_id = "m" } + end + + it("keeps and marks the current round across " .. role, function() + for _, placement in ipairs({ 3, 4 }) do + local builder = prompt.new() + builder:add_user("q") + builder:add_assistant("A", { thinking_blocks = block("round") }) + builder:add_function_call("lookup", "{}", "call_1") + builder:add_function_result("lookup", "found", "call_1") + table.insert(builder.messages, placement, transparent_row()) + local consulted = 0 + local dropped, err = builder:drop_stale_thinking(function() + consulted = consulted + 1 + return false + end) + test.is_nil(err) + test.eq(dropped, builder) + test.eq(thinking_of(builder.messages, 2), "round", "current thinking survives " .. role) + test.eq(consulted, 0) + builder:mark_stale_thinking(function() return false end) + test.is_true(builder.messages[2].metadata.stale_thinking) + test.eq(thinking_of(builder.messages, 2), "round") + end + end) + + for _, carrier in ipairs({ "before", "after" }) do + it("protects an open call with thinking " .. carrier .. " " .. role, function() + local builder = prompt.new() + builder:add_user("q") + builder:add_assistant("A", carrier == "before" and { thinking_blocks = block("round") } or nil) + builder:add_function_call("lookup", "{}", "call_open") + builder:add_function_call("fetch", "{}", "call_done") + if carrier == "after" then builder.messages[4].metadata = { thinking_blocks = block("round") } end + builder:add_function_result("fetch", "found", "call_done") + builder:add_assistant("later") + builder:add_user("next") + table.insert(builder.messages, carrier == "before" and 3 or 4, transparent_row()) + local dropped, err = builder:drop_stale_thinking(function() return false end) + test.is_nil(dropped, "open call across " .. role) + test.contains(tostring(err), "call_open") + test.eq(thinking_of(builder.messages, carrier == "before" and 2 or 5), "round") + end) + end + end + it("keeps the round when a user message joins its tool-result turn", function() local builder = current_round() builder:add_user("Use metric units") diff --git a/src/llm/test/replay_safety_test.lua b/src/llm/test/replay_safety_test.lua new file mode 100644 index 0000000..94dffd6 --- /dev/null +++ b/src/llm/test/replay_safety_test.lua @@ -0,0 +1,1600 @@ +local test = require("test") +local json = require("json") +local registry = require("registry") +local llm = require("llm") +local prompt = require("prompt") +local providers = require("providers") +local thinking = require("thinking") +local claude_mapper = require("claude_mapper") +local claude_client = require("claude_client") +local claude_generate = require("claude_generate") +local claude_structured = require("claude_structured") +local bedrock_mapper = require("bedrock_mapper") +local bedrock_client = require("bedrock_client") +local bedrock_generate = require("bedrock_generate") +local bedrock_structured = require("bedrock_structured") +local openai_mapper = require("openai_mapper") +local openai_generate = require("openai_generate") +local openai_structured = require("openai_structured") +local compat_mapper = require("compat_mapper") +local compat_generate = require("compat_generate") +local compat_structured = require("compat_structured") +local google_mapper = require("google_mapper") +local google_generate = require("google_generate") +local google_structured = require("google_structured") + +local BINDING_BETA = "thinking-binding-controls-2026-08-01" +local SCHEMA = { type = "object", properties = { a = { type = "number" } }, required = { "a" }, additionalProperties = false } + +local function text(value) return { { type = "text", text = value } } end +local function signed(name) return { type = "thinking", thinking = "", signature = name } end + +local CLAUDE_REPLY = { content = { { type = "text", text = "{\"a\":1}" }, + { type = "tool_use", name = "structured_output", id = "s", input = { a = 1 } } }, + stop_reason = "end_turn", usage = { input_tokens = 1, output_tokens = 1 } } +local BEDROCK_REPLY = { output = { message = { role = "assistant", content = { { text = "ok" }, + { toolUse = { toolUseId = "s", name = "structured_output", input = { a = 1 } } } } } }, + stopReason = "end_turn", usage = { inputTokens = 1, outputTokens = 1 } } + +local function capture_claude(handler): any + local sent: any = {} + handler._client = { ENDPOINTS = { MESSAGES = "/messages" }, request = function(_, payload, options) + table.insert(sent, { payload = payload, betas = options and options.beta_features }) + return json.decode((json.encode(CLAUDE_REPLY))) + end } + return sent +end + +local function capture_bedrock(handler): any + local sent: any = {} + handler._client = { converse = function(_, payload) + table.insert(sent, { payload = payload }) + return json.decode((json.encode(BEDROCK_REPLY))) + end } + return sent +end + +local function capture_google(handler): any + local sent: any = {} + local client_contract = { with_context = function(self) return self end, + open = function(self) return self end, + request = function(_, args) + table.insert(sent, { payload = args.payload or args }) + return { candidates = { { content = { parts = { { text = "{\"a\":1}" } } }, finishReason = "STOP" } }, + usageMetadata = { promptTokenCount = 1, candidatesTokenCount = 1 } } + end } + handler._contract = { get = function() return client_contract end } + handler._ctx = { all = function() return {} end, get = function() return "client" end } + return sent +end + +local function capture_openai(handler): any + local sent: any = {} + handler._client = { ENDPOINTS = { RESPONSES = "/responses", CHAT_COMPLETIONS = "/chat/completions" }, + request = function(_, payload) + table.insert(sent, { payload = payload }) + return { status = "completed", output = { { type = "message", content = { { type = "output_text", text = "{\"a\":1}" } } } }, + choices = { { message = { role = "assistant", content = "{\"a\":1}" }, finish_reason = "stop" } }, + usage = { input_tokens = 1, output_tokens = 1, prompt_tokens = 1, completion_tokens = 1 }, metadata = {} } + end } + return sent +end + +-- Wire order of an assistant turn as signatures, call ids and text. +local function claude_order(message) + local order = {} + for _, part in ipairs(message.content) do + table.insert(order, part.signature or part.id or part.text or part.type) + end + return order +end + +local function bedrock_order(message) + local order = {} + for _, part in ipairs(message.content) do + local reasoning = part.reasoningContent and part.reasoningContent.reasoningText + table.insert(order, reasoning and reasoning.signature or (part.toolUse and part.toolUse.toolUseId) + or part.text or (part.cachePoint and "cachePoint") or "other") + end + return order +end + +local function assert_order(actual, expected) + test.eq(table.concat(actual, "|"), table.concat(expected, "|")) +end + +local RESPONSE_ORDER = { "s1", "before ", "s2", "c1", "s3", "c2", "after" } + +local CLAUDE_RESPONSE = { + { type = "thinking", thinking = "", signature = "s1" }, + { type = "text", text = "before " }, + { type = "thinking", thinking = "", signature = "s2" }, + { type = "tool_use", id = "c1", name = "t", input = { a = 1 } }, + { type = "thinking", thinking = "", signature = "s3" }, + { type = "tool_use", id = "c2", name = "t", input = { a = 2 } }, + { type = "text", text = "after" }, +} + +local function reasoning_block(name) return { reasoningContent = { reasoningText = { text = "", signature = name } } } end + +local BEDROCK_RESPONSE = { + reasoning_block("s1"), + { text = "before " }, + reasoning_block("s2"), + { toolUse = { toolUseId = "c1", name = "t", input = { a = 1 } } }, + reasoning_block("s3"), + { toolUse = { toolUseId = "c2", name = "t", input = { a = 2 } } }, + { text = "after" }, +} + +local function sse_stream(events) + local body = "" + for _, event in ipairs(events) do + body = body .. "event: " .. event[1] .. "\ndata: " .. json.encode(event[2]) .. "\n\n" + end + local done = false + return { stream = { read = function() + if done then return nil end + done = true + return body + end }, metadata = {} } +end + +local function claude_stream_events(response, unclosed, stop_reason) + local events = { { "message_start", { message = { usage = { input_tokens = 1 } } } } } + for index, block in ipairs(response) do + local at = index - 1 + if block.type == "thinking" then + table.insert(events, { "content_block_start", { index = at, content_block = { type = "thinking", thinking = "", signature = "" } } }) + table.insert(events, { "content_block_delta", { index = at, delta = { type = "thinking_delta", thinking = block.thinking } } }) + if block.signature ~= "" then + table.insert(events, { "content_block_delta", { index = at, delta = { type = "signature_delta", signature = block.signature } } }) + end + elseif block.type == "text" then + table.insert(events, { "content_block_start", { index = at, content_block = { type = "text", text = "" } } }) + table.insert(events, { "content_block_delta", { index = at, delta = { type = "text_delta", text = block.text } } }) + else + table.insert(events, { "content_block_start", { index = at, content_block = { type = "tool_use", id = block.id, name = block.name } } }) + table.insert(events, { "content_block_delta", { index = at, delta = { type = "input_json_delta", partial_json = json.encode(block.input) } } }) + end + if not (unclosed and index == #response) then + table.insert(events, { "content_block_stop", { index = at } }) + end + end + table.insert(events, { "message_delta", { delta = { stop_reason = stop_reason or (unclosed and "max_tokens" or "tool_use") }, usage = { output_tokens = 2 } } }) + table.insert(events, { "message_stop", {} }) + return events +end + +local function eventstream_message(event_type, body) + local function header(name, value) + return string.char(#name) .. name .. string.char(7) .. string.pack(">I2", #value) .. value + end + local headers = header(":event-type", event_type) .. header(":content-type", "application/json") + .. header(":message-type", "event") + local payload = json.encode(body) + local total = 12 + #headers + #payload + 4 + return string.pack(">I4I4I4", total, #headers, 0) .. headers .. payload .. string.pack(">I4", 0) +end + +local function bedrock_stream(response, unclosed, stop_reason) + local chunks = { eventstream_message("messageStart", { role = "assistant" }) } + local function add(name, body) table.insert(chunks, eventstream_message(name, body)) end + for index, block in ipairs(response) do + local at = index - 1 + if block.reasoningContent then + local reasoning = block.reasoningContent.reasoningText + add("contentBlockDelta", { contentBlockIndex = at, delta = { reasoningContent = { text = reasoning.text } } }) + if reasoning.signature ~= "" then + add("contentBlockDelta", { contentBlockIndex = at, delta = { reasoningContent = { signature = reasoning.signature } } }) + end + elseif block.text then + add("contentBlockDelta", { contentBlockIndex = at, delta = { text = block.text } }) + else + add("contentBlockStart", { contentBlockIndex = at, start = { toolUse = { toolUseId = block.toolUse.toolUseId, name = "t" } } }) + add("contentBlockDelta", { contentBlockIndex = at, delta = { toolUse = { input = json.encode(block.toolUse.input) } } }) + end + if not (unclosed and index == #response) then + add("contentBlockStop", { contentBlockIndex = at }) + end + end + add("messageStop", { stopReason = stop_reason or (unclosed and "max_tokens" or "tool_use") }) + add("metadata", { usage = { inputTokens = 1, outputTokens = 2 } }) + local at = 0 + return { stream = { read = function() + at = at + 1 + return chunks[at] + end }, metadata = {} } +end + +-- The rows a host persists from one response. +local function stored_rows(result): any + local metadata = { + thinking_blocks = result.metadata.thinking_blocks, + content_positions = result.metadata.content_positions, + } + local rows = { + { role = "user", content = text("q") }, + { role = "assistant", content = text(result.result.content), metadata = metadata }, + } + for _, call in ipairs(result.result.tool_calls) do + table.insert(rows, { role = "function_call", content = {}, + function_call = { id = call.id, name = call.name, arguments = call.arguments } }) + end + for _, call in ipairs(result.result.tool_calls) do + table.insert(rows, { role = "function_result", name = call.name, function_call_id = call.id, content = text("r") }) + end + return rows +end + +local function claude_replay(rows, options) + local sent = capture_claude(claude_generate) + local _, err = claude_generate.handler({ model = "m", messages = rows, options = options or { max_tokens = 100 } }) + test.is_nil(err) + return sent[1].payload, sent[1].betas +end + +local function bedrock_replay(rows, options) + local sent = capture_bedrock(bedrock_generate) + local _, err = bedrock_generate.handler({ model = "m", messages = rows, options = options or { max_tokens = 100 } }) + test.is_nil(err) + return sent[1].payload +end + +local function claude_replay_order(result) + return claude_order(claude_replay(stored_rows(result)).messages[2]) +end + +local function bedrock_replay_order(result) + return bedrock_order(bedrock_replay(stored_rows(result)).messages[2]) +end + +local function has_block(list, predicate) + for _, message in ipairs(list) do + if type(message.content) == "table" then + for _, part in ipairs(message.content) do + if predicate(part) then return true end + end + end + end + return false +end + +local function stale_rows(): any + return { + { role = "user", content = text("q") }, + { role = "assistant", content = text("handing off"), metadata = { + thinking_blocks = { signed("s1") }, stale_thinking = true } }, + { role = "function_call", content = {}, function_call = { id = "c1", name = "handoff", arguments = {} } }, + { role = "function_result", name = "handoff", function_call_id = "c1", content = text("ok") }, + } +end + +local function is_claude_thinking(part) return part.type == "thinking" end +local function is_bedrock_reasoning(part) return part.reasoningContent ~= nil end + +local function policy_providers(handler, method, policy) + return { + driver_declares_legacy_reasoning_flag = function() return false end, + driver_output_policy = function() return policy, nil end, + open = function() + return { [method] = function(_, args) return handler.handler(args) end } + end, + } +end + +local function use_card(card) + llm._models = { get_by_name = function() return card, nil end, get_by_class = function() return {}, nil end } +end + +local BUDGET_POLICY = { thinking = "budget", default_max_tokens = 2000 } + +-- A response as the wire blocks of each driver, from { "s1", "text", { "c1" } } shorthand. +local function claude_blocks(shape) + local blocks = {} + for _, item in ipairs(shape) do + if type(item) == "table" and item.type == "thinking" then + table.insert(blocks, item) + elseif type(item) == "table" then + table.insert(blocks, { type = "tool_use", id = item[1], name = "t", input = { a = 1 } }) + elseif item:match("^s%d$") then + table.insert(blocks, { type = "thinking", thinking = "", signature = item }) + else + table.insert(blocks, { type = "text", text = item }) + end + end + return blocks +end + +local function bedrock_blocks(shape) + local blocks = {} + for _, item in ipairs(shape) do + if type(item) == "table" and item.type == "thinking" then + table.insert(blocks, { reasoningContent = { reasoningText = { text = item.thinking, signature = item.signature } } }) + elseif type(item) == "table" then + table.insert(blocks, { toolUse = { toolUseId = item[1], name = "t", input = { a = 1 } } }) + elseif item:match("^s%d$") then + table.insert(blocks, reasoning_block(item)) + else + table.insert(blocks, { text = item }) + end + end + return blocks +end + +-- The four extraction paths, each returning a response as llm hands it to a host. +local EXTRACTION_PATHS = { + { name = "Claude JSON", replay = claude_replay, order = claude_order, extract = function(shape, _, stop_reason) + return claude_mapper.format_success_response({ content = claude_blocks(shape), stop_reason = stop_reason or "tool_use", + usage = { input_tokens = 1, output_tokens = 1 } }, "m", {}) + end }, + { name = "Claude stream", streaming = true, replay = claude_replay, order = claude_order, extract = function(shape, unclosed, stop_reason) + local _, err, stream = claude_client.process_stream(sse_stream(claude_stream_events(claude_blocks(shape), unclosed, stop_reason)), {}) + test.is_nil(err) + return claude_mapper.format_streaming_response(stream, {}, stream.usage, stream.finish_reason, stream.metadata) + end }, + { name = "Bedrock JSON", replay = bedrock_replay, order = bedrock_order, extract = function(shape, _, stop_reason) + return bedrock_mapper.format_success_response({ output = { message = { role = "assistant", content = bedrock_blocks(shape) } }, + stopReason = stop_reason or "tool_use", usage = { inputTokens = 1, outputTokens = 1 } }, {}) + end }, + { name = "Bedrock stream", streaming = true, replay = bedrock_replay, order = bedrock_order, extract = function(shape, unclosed, stop_reason) + local _, err, stream = bedrock_client.process_converse_stream(bedrock_stream(bedrock_blocks(shape), unclosed, stop_reason), {}) + test.is_nil(err) + return bedrock_mapper.format_streaming_response(stream, {}, stream.usage, stream.finish_reason, stream.metadata) + end }, +} + +local function define_tests() + describe("Replay safety", function() + after_each(function() + llm._providers = nil + llm._models = nil + llm._usage_tracker = nil + end) + + it("preserves caller rows when compat maps before a Claude fallback", function() + local rows = { + { role = "system", content = text("Base") }, + { role = "user", content = text("Question") }, + { role = "system", content = text("Turn context"), metadata = { preserve_position = true } }, + { role = "assistant", content = text("Answer"), metadata = { + thinking_blocks = { { type = "thinking", thinking = "Reasoning", signature = "sig" }, + { type = "redacted_thinking", data = "opaque" } }, + } }, + { role = "user", content = text("Follow-up") }, + { role = "developer", content = text("Recall"), metadata = { + preserve_position = true, memory_ids = { "memory-1", "memory-2" }, + } }, + } + local before = json.encode(rows) + + local compat = compat_mapper.map_messages(rows) + local after_compat = json.encode(rows) + local replay = claude_mapper.map_messages(rows) + + test.eq(replay.messages[2].content[1].type, "thinking", "Claude fallback keeps thinking blocks") + test.eq(replay.messages[2].content[1].signature, "sig") + test.eq(replay.messages[2].content[2].type, "redacted_thinking") + test.eq(replay.messages[2].content[2].data, "opaque") + test.eq(replay.messages[2].content[3].text, "Answer") + test.eq(#replay.system, 1) + test.eq(replay.system[1].text, "Base") + test.eq(replay.messages[1].content[2].text, "Turn context") + test.eq(replay.messages[3].content[2].text, "Recall") + test.eq(after_compat, before, "compat leaves all input tables unchanged") + test.eq(json.encode(rows), before, "Claude leaves all input tables unchanged") + for _, row in ipairs(compat) do test.is_nil(row.metadata) end + end) + + describe("stale blocks without a bindable thinking object", function() + local UNBINDABLE = { + { name = "default budget at effort 0", accepts = nil, options = { max_tokens = 100 } }, + { name = "adaptive at effort 0", accepts = { thinking = "adaptive" }, options = { max_tokens = 100, thinking_effort = 0 } }, + { name = "none", accepts = { thinking = "none" }, options = { max_tokens = 100 } }, + } + for _, case in ipairs(UNBINDABLE) do + it("removes marked blocks on Claude: " .. case.name, function() + local sent = capture_claude(claude_generate) + assert(claude_generate.handler({ model = "m", messages = stale_rows(), options = case.options, accepts = case.accepts })) + test.is_false(has_block(sent[1].payload.messages, is_claude_thinking)) + test.is_nil(sent[1].betas) + assert_order(claude_order(sent[1].payload.messages[2]), { "handing off", "c1" }) + end) + it("removes marked blocks on Bedrock: " .. case.name, function() + local sent = capture_bedrock(bedrock_generate) + assert(bedrock_generate.handler({ model = "m", messages = stale_rows(), options = case.options, accepts = case.accepts })) + test.is_false(has_block(sent[1].payload.messages, is_bedrock_reasoning)) + assert_order(bedrock_order(sent[1].payload.messages[2]), { "handing off", "c1" }) + end) + end + + it("removes marked blocks on Claude between_tools", function() + local sent = capture_claude(claude_generate) + assert(claude_generate.handler({ model = "m", messages = stale_rows(), options = { + max_tokens = 100, thinking_effort = 50, request = { thinking = { type = "between_tools" } } }, + accepts = { thinking = "adaptive" } })) + test.eq(sent[1].payload.thinking.type, "between_tools") + test.is_false(has_block(sent[1].payload.messages, is_claude_thinking)) + end) + + it("removes marked blocks on Bedrock between_tools", function() + local sent = capture_bedrock(bedrock_generate) + assert(bedrock_generate.handler({ model = "m", messages = stale_rows(), options = { + max_tokens = 100, thinking_effort = 50, + request = { additionalModelRequestFields = { thinking = { type = "between_tools" } } } }, + accepts = { thinking = "adaptive" } })) + test.eq(sent[1].payload.additionalModelRequestFields.thinking.type, "between_tools") + test.is_false(has_block(sent[1].payload.messages, is_bedrock_reasoning)) + end) + + it("removes marked blocks in Claude and Bedrock structured output", function() + local sent = capture_claude(claude_structured) + assert(claude_structured.handler({ model = "m", messages = stale_rows(), schema = SCHEMA, options = { max_tokens = 100 } })) + test.is_false(has_block(sent[1].payload.messages, is_claude_thinking)) + sent = capture_claude(claude_structured) + assert(claude_structured.handler({ model = "m", messages = stale_rows(), schema = SCHEMA, + options = { max_tokens = 100 }, accepts = { structured_output = "native" } })) + test.is_false(has_block(sent[1].payload.messages, is_claude_thinking)) + sent = capture_bedrock(bedrock_structured) + assert(bedrock_structured.handler({ model = "m", messages = stale_rows(), schema = SCHEMA, options = { max_tokens = 100 } })) + test.is_false(has_block(sent[1].payload.messages, is_bedrock_reasoning)) + end) + + local BINDABLE = { + { name = "always at effort 0", accepts = { thinking = "always" }, options = { max_tokens = 100 } }, + { name = "adaptive above 0", accepts = { thinking = "adaptive" }, options = { max_tokens = 100, thinking_effort = 30 } }, + { name = "budget above 0", accepts = { thinking = "budget" }, options = { max_tokens = 100, thinking_effort = 30 } }, + } + for _, case in ipairs(BINDABLE) do + it("keeps marked blocks bound on Claude: " .. case.name, function() + local sent = capture_claude(claude_generate) + assert(claude_generate.handler({ model = "m", messages = stale_rows(), options = case.options, accepts = case.accepts })) + test.is_true(has_block(sent[1].payload.messages, is_claude_thinking)) + test.eq(sent[1].payload.thinking.block_binding.prefix_mismatch_behavior, "drop_block") + test.eq(sent[1].betas[1], BINDING_BETA) + end) + it("keeps marked blocks bound on Bedrock: " .. case.name, function() + local sent = capture_bedrock(bedrock_generate) + assert(bedrock_generate.handler({ model = "m", messages = stale_rows(), options = case.options, accepts = case.accepts })) + test.is_true(has_block(sent[1].payload.messages, is_bedrock_reasoning)) + test.eq(sent[1].payload.additionalModelRequestFields.thinking.block_binding.prefix_mismatch_behavior, "drop_block") + end) + end + + it("decides from the object after the overlay", function() + local sent = capture_claude(claude_generate) + assert(claude_generate.handler({ model = "m", messages = stale_rows(), accepts = { thinking = "adaptive" }, options = { + max_tokens = 100, thinking_effort = 30, request = { thinking = { type = "disabled" } } } })) + test.is_false(has_block(sent[1].payload.messages, is_claude_thinking)) + + sent = capture_claude(claude_generate) + assert(claude_generate.handler({ model = "m", messages = stale_rows(), accepts = { thinking = "none" }, options = { + max_tokens = 100, request = { thinking = { type = "adaptive" } } } })) + test.is_true(has_block(sent[1].payload.messages, is_claude_thinking)) + test.eq(sent[1].payload.thinking.block_binding.prefix_mismatch_behavior, "drop_block") + end) + + it("leaves unmarked blocks on a request without a thinking object", function() + local rows = stale_rows() + rows[2].metadata.stale_thinking = nil + local payload = claude_replay(rows) + test.is_true(has_block(payload.messages, is_claude_thinking)) + payload = bedrock_replay(stale_rows()) + test.is_false(has_block(payload.messages, is_bedrock_reasoning)) + end) + end) + + describe("interleaved response order", function() + it("replays Claude JSON exactly", function() + local result = claude_mapper.format_success_response({ content = CLAUDE_RESPONSE, stop_reason = "tool_use", + usage = { input_tokens = 1, output_tokens = 1 } }, "m", {}) + assert_order(claude_replay_order({ result = result.result, metadata = result.metadata }), RESPONSE_ORDER) + end) + + it("replays Claude stream exactly", function() + local _, stream_err, stream_result = claude_client.process_stream(sse_stream(claude_stream_events(CLAUDE_RESPONSE)), {}) + test.is_nil(stream_err) + local result = claude_mapper.format_streaming_response(stream_result, {}, stream_result.usage, stream_result.finish_reason, stream_result.metadata) + assert_order(claude_replay_order(result), RESPONSE_ORDER) + end) + + it("replays Bedrock JSON exactly", function() + local result = bedrock_mapper.format_success_response({ output = { message = { role = "assistant", content = BEDROCK_RESPONSE } }, + stopReason = "tool_use", usage = { inputTokens = 1, outputTokens = 1 } }, {}) + assert_order(bedrock_replay_order(result), RESPONSE_ORDER) + end) + + it("replays Bedrock stream exactly", function() + local _, stream_err, stream_result = bedrock_client.process_converse_stream(bedrock_stream(BEDROCK_RESPONSE), {}) + test.is_nil(stream_err) + local result = bedrock_mapper.format_streaming_response(stream_result, {}, stream_result.usage, stream_result.finish_reason, stream_result.metadata) + assert_order(bedrock_replay_order(result), RESPONSE_ORDER) + end) + + for _, path in ipairs(EXTRACTION_PATHS) do + it("keeps an edited response out of the preceding layout: " .. path.name, function() + local rows = stored_rows(path.extract({ "a", "s1", "b" }, false, "end_turn")) + local second = stored_rows(path.extract({ "s2", "c", "d" }, false, "end_turn"))[2] + test.not_nil(rows[2].metadata.content_positions) + test.not_nil(second.metadata.content_positions) + second.content = text("edited") + table.insert(rows, second) + local payload = path.replay(rows) + test.eq(#payload.messages, 2) + assert_order(path.order(payload.messages[2]), { "a", "s1", "b", "s2", "edited" }) + end) + + it("renders no blank layout text beside visible text: " .. path.name, function() + local result: any = path.extract({ "s1", "A", { "c1" }, "s2", "\n\n", { "c2" } }) + test.eq(result.result.content, "A\n\n") + local payload = path.replay(stored_rows(result)) + assert_order(path.order(payload.messages[2]), { "s1", "A", "c1", "s2", "c2" }) + end) + + for label, stored in pairs({ + ["as received"] = " \n", ["as empty text"] = "", ["as other whitespace"] = "\t", + }) do + it("keeps the order of a layout with only blank text stored " .. label .. ": " .. path.name, function() + local result: any = path.extract({ "s1", " ", { "c1" }, "s2", "\n", { "c2" } }) + local rows = stored_rows(result) + rows[2].content = text(stored) + local payload = path.replay(rows) + assert_order(path.order(payload.messages[2]), { "s1", "c1", "s2", "c2" }) + end) + end + + it("keeps the order of a layout with only blank text stored with no content: " .. path.name, function() + local result: any = path.extract({ "s1", " ", { "c1" }, "s2", "\n", { "c2" } }) + local rows = stored_rows(result) + rows[2].content = {} + local payload = path.replay(rows) + assert_order(path.order(payload.messages[2]), { "s1", "c1", "s2", "c2" }) + end) + + it("keeps the layout when the host trimmed the text: " .. path.name, function() + local result: any = path.extract({ "s1", "A", { "c1" }, "s2", "\n\n", { "c2" } }) + local rows = stored_rows(result) + rows[2].content = text("A") + local payload = path.replay(rows) + assert_order(path.order(payload.messages[2]), { "s1", "A", "c1", "s2", "c2" }) + end) + + it("renders no whitespace text of a plain-order response: " .. path.name, function() + local result: any = path.extract({ "s1", "\n\n", { "c1" } }) + test.is_nil(result.metadata.content_positions) + local payload = path.replay(stored_rows(result)) + assert_order(path.order(payload.messages[2]), { "s1", "c1" }) + end) + + it("renders a turn with only blank layout text and no call as nothing: " .. path.name, function() + local result: any = path.extract({ "s1", " ", "s2", "\n" }) + local rows = stored_rows(result) + table.insert(rows, { role = "user", content = text("next") }) + local payload = path.replay(rows) + for _, message in ipairs(payload.messages) do + test.eq(message.role, "user") + end + end) + end + + -- Rows a host stored from a response whose calls it discarded. + local function rows_without_calls(result, content): any + local rows = stored_rows(result) + local kept = { rows[1], rows[2], { role = "user", content = text("next") } } + if content then kept[2].content = content end + return kept + end + + local function assistant_order(path, payload) + for _, message in ipairs(payload.messages) do + if message.role == "assistant" then return path.order(message) end + end + return {} + end + + for _, path in ipairs(EXTRACTION_PATHS) do + for _, keep_second in ipairs({ false, true }) do + it("drops thinking after a missing call before visible text with " .. (keep_second and "one call retained: " or "no calls retained: ") .. path.name, function() + local result: any = path.extract({ "s1", { "c1" }, "s2", "A", { "c2" } }) + local rows = rows_without_calls(result) + if keep_second then + local stored = stored_rows(result) + table.insert(rows, 3, stored[4]) + table.insert(rows, 4, stored[6]) + end + assert_order(assistant_order(path, path.replay(rows)), keep_second and { "s1", "A", "c2" } or { "s1", "A" }) + test.eq(#result.metadata.thinking_blocks, 2) + test.eq(#result.metadata.content_positions, 3) + end) + end + + it("keeps layouts across hoisted systems and ends them at positional notes: " .. path.name, function() + for _, role in ipairs({ "system", "developer" }) do + local rows = stored_rows(path.extract({ "A", "s1", { "c1" } })) + table.insert(rows, 3, { role = role, content = text("NOTE"), metadata = { preserve_position = true } }) + local payload = path.replay(rows) + assert_order(path.order(payload.messages[2]), { "A" }) + assert_order(path.order(payload.messages[4]), { "c1" }) + end + local rows = stored_rows(path.extract({ "A", "s1", { "c1" } })) + table.insert(rows, 3, { role = "system", content = text("SYSTEM") }) + local payload = path.replay(rows) + assert_order(path.order(payload.messages[2]), { "A", "s1", "c1" }) + test.eq(payload.system[1].text, "SYSTEM") + end) + + it("drops thinking behind a missing call whose id belongs to an earlier turn: " .. path.name, function() + local earlier = stored_rows(path.extract({ "before", "s1", { "c" } })) + local before = path.replay(earlier) + local result: any = path.extract({ "A", "s2", { "c" } }, true) + local later = rows_without_calls(result) + local rows = {} + for _, row in ipairs(earlier) do table.insert(rows, row) end + table.insert(rows, later[2]) + table.insert(rows, later[3]) + local payload = path.replay(rows) + for index, message in ipairs(before.messages) do + test.eq(json.encode(payload.messages[index]), json.encode(message), "earlier turn " .. index) + end + local last_assistant + for _, message in ipairs(payload.messages) do + if message.role == "assistant" then last_assistant = message end + end + test.not_nil(last_assistant) + assert_order(path.order(last_assistant), { "A" }) + end) + + it("drops thinking behind a discarded call in a stored row: text, thinking, call: " .. path.name, function() + local result: any = path.extract({ "A", "s1", { "c1" } }) + assert_order(assistant_order(path, path.replay(rows_without_calls(result))), { "A" }) + end) + + it("drops thinking behind a discarded call in a stored row: thinking, call: " .. path.name, function() + local result: any = path.extract({ "s1", { "c1" } }) + assert_order(assistant_order(path, path.replay(rows_without_calls(result))), {}) + end) + + it("keeps thinking in front of the last visible text of a stored row: " .. path.name, function() + local result: any = path.extract({ "s1", "A", "s2", { "c1" } }) + assert_order(assistant_order(path, path.replay(rows_without_calls(result))), { "s1", "A" }) + local rows = rows_without_calls(result) + table.insert(rows, 3, { role = "assistant", content = text("B") }) + assert_order(assistant_order(path, path.replay(rows)), { "s1", "A", "B" }) + end) + + it("drops thinking behind a discarded call when the host dropped the text too: " .. path.name, function() + local result: any = path.extract({ "A", "s1", { "c1" } }) + for _, content in ipairs({ {}, text(""), text(" ") }) do + assert_order(assistant_order(path, path.replay(rows_without_calls(result, content))), {}) + end + end) + + it("keeps the calls that remain and drops thinking behind the last of them: " .. path.name, function() + local result: any = path.extract({ "s1", { "c1" }, "s2", { "c2" } }) + local rows = stored_rows(result) + local kept = {} + for _, row in ipairs(rows) do + local id = row.function_call and row.function_call.id or row.function_call_id + if id ~= "c2" then table.insert(kept, row) end + end + assert_order(assistant_order(path, path.replay(kept)), { "s1", "c1" }) + end) + + for _, case in ipairs({ + { name = "A T", shape = { "A", "s1" }, expected = { "A" } }, + { name = "T A T", shape = { "s1", "A", "s2" }, expected = { "s1", "A" } }, + }) do + it("drops trailing thinking after max_tokens: " .. case.name .. ": " .. path.name, function() + local result: any = path.extract(case.shape, false, "max_tokens") + test.eq(result.finish_reason, "length") + test.eq(#result.result.tool_calls, 0) + test.not_nil(result.metadata.content_positions) + local rows = rows_without_calls(result) + for _ = 1, 2 do + local payload = path.replay(rows) + assert_order(assistant_order(path, payload), case.expected) + test.eq(payload.messages[#payload.messages].role, "user") + table.insert(rows, { role = "assistant", content = text("Later answer") }) + table.insert(rows, { role = "user", content = text("Another question") }) + end + end) + end + + it("omits thinking alone after max_tokens before a user message: " .. path.name, function() + local result: any = path.extract({ "s1" }, false, "max_tokens") + test.eq(result.finish_reason, "length") + test.eq(#result.result.tool_calls, 0) + test.eq(#result.metadata.thinking_blocks, 1) + test.is_nil(result.metadata.content_positions) + local payload = path.replay(rows_without_calls(result)) + for _, message in ipairs(payload.messages) do + test.eq(message.role, "user") + end + test.contains(table.concat(path.order(payload.messages[#payload.messages]), "|"), "next") + end) + + it("keeps signed plain-order thinking in front of text: " .. path.name, function() + local result: any = path.extract({ "s1", "s2", "A" }, false, "max_tokens") + test.eq(result.finish_reason, "length") + test.is_nil(result.metadata.content_positions) + for _, block in ipairs(result.metadata.thinking_blocks) do test.is_nil(block.position) end + assert_order(assistant_order(path, path.replay(rows_without_calls(result))), { "s1", "s2", "A" }) + end) + + it("keeps a completed call turn after max_tokens: " .. path.name, function() + local result: any = path.extract({ "s1", "A", "s2", { "c1" } }, false, "max_tokens") + test.eq(result.finish_reason, "length") + test.eq(#result.result.tool_calls, 1) + local payload = path.replay(stored_rows(result)) + assert_order(assistant_order(path, payload), { "s1", "A", "s2", "c1" }) + test.eq(payload.messages[#payload.messages].role, "user") + end) + + for _, layout in ipairs({ false, true }) do + it("drops an empty thinking signature before text in " .. (layout and "a layout" or "plain order") .. ": " .. path.name, function() + local unsigned = { type = "thinking", thinking = "unfinished", signature = "" } + local shape = layout and { "before ", unsigned, "A" } or { unsigned, "A" } + local result: any = path.extract(shape, false, "max_tokens") + test.eq(result.metadata.thinking_blocks[1].signature, "") + test.eq(result.metadata.content_positions ~= nil, layout) + local payload = path.replay(rows_without_calls(result)) + assert_order(assistant_order(path, payload), layout and { "before ", "A" } or { "A" }) + end) + end + + if path.streaming then + it("drops an unsigned block cut mid-thinking after text: " .. path.name, function() + local result: any = path.extract({ "A", { type = "thinking", thinking = "unfinished", signature = "" } }, true) + test.eq(result.finish_reason, "length") + test.eq(#result.result.tool_calls, 0) + test.eq(result.metadata.thinking_blocks[1].thinking, "unfinished") + test.eq(result.metadata.thinking_blocks[1].signature, "") + local payload = path.replay(rows_without_calls(result)) + assert_order(assistant_order(path, payload), { "A" }) + end) + end + + it("ends each split response turn with text or a call: " .. path.name, function() + local result: any = path.extract({ "s1", "A", { "c1" }, "s2", { "c2" } }) + local rows = stored_rows(result) + local paired = { rows[1], rows[2], rows[3], rows[5], rows[4], rows[6] } + local payload = path.replay(paired) + local blocks = path.order == claude_order and claude_blocks or bedrock_blocks + test.eq(#payload.messages, 5) + for index, role in ipairs({ "user", "assistant", "user", "assistant", "user" }) do + test.eq(payload.messages[index].role, role) + end + test.eq(json.encode(payload.messages[2].content), json.encode(blocks({ "s1", "A", { "c1" } })), + "first emitted turn") + test.eq(json.encode(payload.messages[4].content), json.encode(blocks({ { "c2" } })), + "second emitted turn") + end) + + it("ends a turn before thinking-only rows without positions: " .. path.name, function() + for _, redacted in ipairs({ false, true }) do + for _, cached in ipairs({ false, true }) do + local result: any = path.extract({ "s1", "A", { "c1" } }) + local rows = stored_rows(result) + table.insert(rows, 4, { role = "assistant", content = {}, metadata = { + thinking_blocks = { redacted and { type = "redacted_thinking", data = "opaque" } or signed("s2") } } }) + if cached then table.insert(rows, 5, { role = "cache_marker", marker_id = "tail" }) end + local payload = path.replay(rows) + local blocks = path.order == claude_order and claude_blocks or bedrock_blocks + local expected = blocks({ "s1", "A", { "c1" } }) + if cached then + if path.order == claude_order then + expected[3].cache_control = { type = "ephemeral" } + else + table.insert(expected, { cachePoint = { type = "default" } }) + end + end + test.eq(#payload.messages, 3) + test.eq(json.encode(payload.messages[2].content), json.encode(expected), "emitted turn without positions") + if path.order == claude_order then + rows[4].content = rows[4].metadata.thinking_blocks + rows[4].metadata = nil + payload = path.replay(rows) + test.eq(json.encode(payload.messages[2].content), json.encode(expected), "emitted turn with inline thinking") + end + end + end + end) + end + + it("treats a streamed call that never closed as discarded", function() + local events = claude_stream_events(claude_blocks({ "A", "s1", { "c1" } })) + local kept = {} + for _, event in ipairs(events) do + if not (event[1] == "content_block_stop" and event[2].index == 2) then + table.insert(kept, event) + end + end + local _, _, stream = claude_client.process_stream(sse_stream(kept), {}) + local result: any = claude_mapper.format_streaming_response(stream, {}, stream.usage, "max_tokens", stream.metadata) + test.eq(#result.result.tool_calls, 0) + local payload = claude_replay({ { role = "user", content = text("q") }, + { role = "assistant", content = text(result.result.content), metadata = { + thinking_blocks = result.metadata.thinking_blocks, + content_positions = result.metadata.content_positions } }, + { role = "user", content = text("next") } }) + assert_order(claude_order(payload.messages[2]), { "A" }) + end) + + it("sends no position field on the wire", function() + local result = claude_mapper.format_success_response({ content = CLAUDE_RESPONSE, stop_reason = "tool_use", + usage = { input_tokens = 1, output_tokens = 1 } }, "m", {}) + local payload = claude_replay(stored_rows(result)) + test.is_false(json.encode(payload):find("position", 1, true) ~= nil) + end) + + it("records nothing for a response the plain order reproduces", function() + local result = claude_mapper.format_success_response({ content = { + { type = "thinking", thinking = "", signature = "s1" }, + { type = "text", text = "only" }, + { type = "tool_use", id = "c1", name = "t", input = { a = 1 } }, + }, stop_reason = "tool_use", usage = { input_tokens = 1, output_tokens = 1 } }, "m", {}) + test.is_nil(result.metadata.content_positions) + test.is_nil(result.metadata.thinking_blocks[1].position) + end) + + it("keeps today's order for stored rows without positions", function() + local rows = { + { role = "user", content = text("q") }, + { role = "assistant", content = text("before after"), metadata = { + thinking_blocks = { signed("s1"), signed("s2"), signed("s3") } } }, + { role = "function_call", content = {}, function_call = { id = "c1", name = "t", arguments = {} } }, + { role = "function_call", content = {}, function_call = { id = "c2", name = "t", arguments = {} } }, + { role = "function_result", name = "t", function_call_id = "c1", content = text("r") }, + { role = "function_result", name = "t", function_call_id = "c2", content = text("r") }, + } + assert_order(claude_order(claude_replay(rows).messages[2]), { "s1", "s2", "s3", "before after", "c1", "c2" }) + assert_order(bedrock_order(bedrock_replay(rows).messages[2]), { "s1", "s2", "s3", "before after", "c1", "c2" }) + end) + + it("keeps the text and call order once the blocks are stripped", function() + local result = claude_mapper.format_success_response({ content = CLAUDE_RESPONSE, stop_reason = "tool_use", + usage = { input_tokens = 1, output_tokens = 1 } }, "m", {}) + local rows = stored_rows({ result = result.result, metadata = result.metadata }) + rows[2].metadata.thinking_blocks = nil + assert_order(claude_order(claude_replay(rows).messages[2]), { "before ", "c1", "c2", "after" }) + end) + + it("keeps the row's text when the host changed it", function() + local result = claude_mapper.format_success_response({ content = CLAUDE_RESPONSE, stop_reason = "tool_use", + usage = { input_tokens = 1, output_tokens = 1 } }, "m", {}) + local rows = stored_rows({ result = result.result, metadata = result.metadata }) + rows[2].content = text("edited") + assert_order(claude_order(claude_replay(rows).messages[2]), { "s1", "s2", "s3", "edited", "c1", "c2" }) + end) + + it("keeps two responses of one consolidated turn apart", function() + local first = claude_mapper.format_success_response({ content = { + { type = "text", text = "a" }, { type = "thinking", thinking = "", signature = "s1" }, { type = "text", text = "b" }, + }, stop_reason = "end_turn", usage = { input_tokens = 1, output_tokens = 1 } }, "m", {}) + local second = claude_mapper.format_success_response({ content = { + { type = "text", text = "c" }, { type = "thinking", thinking = "", signature = "s2" }, { type = "text", text = "d" }, + }, stop_reason = "end_turn", usage = { input_tokens = 1, output_tokens = 1 } }, "m", {}) + local function row(result) + return { role = "assistant", content = text(result.result.content), metadata = { + thinking_blocks = result.metadata.thinking_blocks, content_positions = result.metadata.content_positions } } + end + local rows = { { role = "user", content = text("q") }, row(first), row(second), { role = "user", content = text("next") } } + assert_order(claude_order(claude_replay(rows).messages[2]), { "a", "s1", "b", "c", "s2", "d" }) + end) + + it("keeps a Bedrock cache point behind the block it follows", function() + local result = bedrock_mapper.format_success_response({ output = { message = { role = "assistant", content = BEDROCK_RESPONSE } }, + stopReason = "tool_use", usage = { inputTokens = 1, outputTokens = 1 } }, {}) + local rows = stored_rows(result) + table.insert(rows, 3, { role = "cache_marker", marker_id = "m" }) + assert_order(bedrock_order(bedrock_replay(rows).messages[2]), + { "s1", "before ", "s2", "c1", "s3", "c2", "after", "cachePoint" }) + end) + + it("takes no text layout from a recovered text tool call", function() + local extracted = bedrock_mapper.extract_response_content({ output = { message = { content = { + reasoning_block("s1"), { text = "a" }, { text = '{"name":"t","arguments":{}}' }, + } } } }, { t = "app:t" }) + test.eq(#extracted.tool_calls, 1) + test.is_nil(extracted.content_positions) + end) + end) + + describe("OpenAI-compatible tool rounds", function() + for _, role in ipairs({ "system", "developer" }) do + it("emits an intervening " .. role .. " note after all tool results", function() + local mapped: any = compat_mapper.map_messages({ + { role = "assistant", content = text("look") }, + { role = "function_call", content = {}, function_call = { id = "c1", name = "t", arguments = "{}" } }, + { role = "function_call", content = {}, function_call = { id = "c2", name = "t", arguments = "{}" } }, + { role = "function_result", name = "t", function_call_id = "c1", content = text("r1") }, + { role = role, content = text("NOTE"), metadata = { preserve_position = true } }, + { role = "function_result", name = "t", function_call_id = "c2", content = text("r2") }, + }, { model = "gpt" }) + local roles = {} + for _, message in ipairs(mapped) do table.insert(roles, message.role) end + assert_order(roles, { "assistant", "tool", "tool", "system" }) + test.eq(#mapped[1].tool_calls, 2) + test.eq(mapped[1].tool_calls[1].id, "c1") + test.eq(mapped[1].tool_calls[2].id, "c2") + test.eq(mapped[2].tool_call_id, "c1") + test.eq(mapped[2].content, "r1") + test.eq(mapped[3].tool_call_id, "c2") + test.eq(mapped[3].content, "r2") + test.eq(mapped[4].content[1].text, "NOTE") + end) + end + + local function note_rows() + return { + { role = "user", content = text("q") }, + { role = "assistant", content = text("look") }, + { role = "function_call", content = {}, function_call = { id = "c1", name = "t", arguments = "{}" } }, + { role = "function_result", name = "t", function_call_id = "c1", content = text("r1") }, + { role = "system", content = text("NOTE"), metadata = { preserve_position = true } }, + { role = "function_call", content = {}, function_call = { id = "c2", name = "t", arguments = "{}" } }, + { role = "function_result", name = "t", function_call_id = "c2", content = text("r2") }, + } + end + + it("keeps a positional note between tool rounds in place", function() + local mapped: any = compat_mapper.map_messages(note_rows(), { model = "gpt" }) + local roles = {} + for _, message in ipairs(mapped) do table.insert(roles, message.role) end + assert_order(roles, { "user", "assistant", "tool", "system", "assistant", "tool" }) + test.eq(#mapped[2].tool_calls, 1) + test.eq(mapped[2].tool_calls[1].id, "c1") + test.eq(mapped[4].content[1].text, "NOTE") + test.eq(mapped[5].tool_calls[1].id, "c2") + test.eq(mapped[6].tool_call_id, "c2") + end) + + it("keeps parallel calls of one round on one assistant message", function() + local mapped: any = compat_mapper.map_messages({ + { role = "assistant", content = text("look") }, + { role = "function_call", content = {}, function_call = { id = "c1", name = "t", arguments = "{}" } }, + { role = "function_result", name = "t", function_call_id = "c1", content = text("r1") }, + { role = "function_call", content = {}, function_call = { id = "c2", name = "t", arguments = "{}" } }, + { role = "function_result", name = "t", function_call_id = "c2", content = text("r2") }, + }, { model = "gpt" }) + test.eq(#mapped[1].tool_calls, 2) + end) + + it("reads past a cache marker between rounds", function() + local mapped: any = compat_mapper.map_messages({ + { role = "assistant", content = text("look") }, + { role = "function_call", content = {}, function_call = { id = "c1", name = "t", arguments = "{}" } }, + { role = "function_result", name = "t", function_call_id = "c1", content = text("r1") }, + { role = "cache_marker", marker_id = "m" }, + { role = "function_call", content = {}, function_call = { id = "c2", name = "t", arguments = "{}" } }, + { role = "function_result", name = "t", function_call_id = "c2", content = text("r2") }, + }, { model = "gpt" }) + test.eq(#mapped[1].tool_calls, 2) + end) + end) + + describe("blank assistant turns", function() + local function blank_rows(row) + return { { role = "user", content = text("Q") }, row, { role = "user", content = text("Q2") } } + end + local BLANKS = { + { name = "empty text", row = { role = "assistant", content = text("") } }, + { name = "whitespace text", row = { role = "assistant", content = text(" \n\t") } }, + { name = "no content", row = { role = "assistant", content = {} } }, + { name = "string blank", row = { role = "assistant", content = " " } }, + { name = "thinking only", row = { role = "assistant", content = {}, metadata = { thinking_blocks = { signed("s1") } } } }, + { name = "thinking and whitespace", row = { role = "assistant", content = text(" "), metadata = { thinking_blocks = { signed("s1") } } } }, + } + for _, case in ipairs(BLANKS) do + it("renders nothing on Claude: " .. case.name, function() + local mapped = claude_mapper.map_messages(blank_rows(case.row)) + test.eq(#mapped.messages, 2) + test.eq(mapped.messages[1].role, "user") + test.eq(mapped.messages[2].role, "user") + test.is_false(json.encode(mapped.messages):find("intercepted", 1, true) ~= nil) + end) + it("renders nothing on Bedrock: " .. case.name, function() + local mapped = bedrock_mapper.map_messages(blank_rows(case.row)) + test.eq(#mapped.messages, 1) + test.eq(mapped.messages[1].role, "user") + test.eq(#mapped.messages[1].content, 2) + end) + it("renders nothing on OpenAI: " .. case.name, function() + local items = openai_mapper.map_messages(blank_rows(case.row), {}) + test.eq(#items, 2) + test.eq(items[1].role, "user") + test.eq(items[2].role, "user") + end) + it("renders nothing on OpenAI-compatible: " .. case.name, function() + local mapped = compat_mapper.map_messages(blank_rows(case.row), {}) + test.eq(#mapped, 2) + test.eq(mapped[2].role, "user") + end) + it("renders nothing on Google: " .. case.name, function() + local mapped = google_mapper.map_messages(blank_rows(case.row), {}) + test.eq(#mapped, 2) + test.eq(mapped[2].role, "user") + end) + end + + it("drops only the blank parts of an OpenAI assistant message", function() + local items = openai_mapper.map_messages({ + { role = "user", content = text("Q") }, + { role = "assistant", content = { { type = "text", text = " " }, { type = "text", text = "kept" }, { type = "text", text = "" } } }, + }, {}) + test.eq(#items[2].content, 1) + test.eq(items[2].content[1].text, "kept") + end) + + it("keeps a turn that has a call on every driver", function() + local rows = { + { role = "user", content = text("Q") }, + { role = "assistant", content = text(""), metadata = { thinking_blocks = { signed("s1") } } }, + { role = "function_call", content = {}, function_call = { id = "c1", name = "t", arguments = "{\"a\":1}" } }, + { role = "function_result", name = "t", function_call_id = "c1", content = text("r") }, + } + assert_order(claude_order(claude_mapper.map_messages(rows).messages[2]), { "s1", "c1" }) + assert_order(bedrock_order(bedrock_mapper.map_messages(rows).messages[2]), { "s1", "c1" }) + test.eq(openai_mapper.map_messages(rows, {})[2].type, "function_call") + test.eq((compat_mapper.map_messages(rows, {}) :: any)[2].tool_calls[1].id, "c1") + test.not_nil(google_mapper.map_messages(rows, {})[2].parts[1].functionCall) + end) + + it("sends no whitespace text inside a turn with a call on any driver", function() + local rows = { + { role = "user", content = text("Q") }, + { role = "assistant", content = text("\n\n") }, + { role = "function_call", content = {}, function_call = { id = "c1", name = "t", arguments = "{}" } }, + { role = "function_result", name = "t", function_call_id = "c1", content = text("r") }, + } + assert_order(claude_order(claude_mapper.map_messages(rows).messages[2]), { "c1" }) + assert_order(bedrock_order(bedrock_mapper.map_messages(rows).messages[2]), { "c1" }) + test.eq(openai_mapper.map_messages(rows, {})[2].type, "function_call") + test.not_nil(google_mapper.map_messages(rows, {})[2].parts[1].functionCall) + test.eq((compat_mapper.map_messages(rows, {}) :: any)[2].content, "") + end) + + it("sends no whitespace string content inside a turn with a call", function() + local rows = { + { role = "user", content = text("Q") }, + { role = "assistant", content = " " }, + { role = "function_call", content = {}, function_call = { id = "c1", name = "t", arguments = "{}" } }, + { role = "function_result", name = "t", function_call_id = "c1", content = text("r") }, + } + assert_order(claude_order(claude_mapper.map_messages(rows).messages[2]), { "c1" }) + assert_order(bedrock_order(bedrock_mapper.map_messages(rows).messages[2]), { "c1" }) + end) + + it("drops a whitespace part beside visible text", function() + local rows = { + { role = "user", content = text("Q") }, + { role = "assistant", content = { { type = "text", text = " " }, { type = "text", text = "kept" } } }, + { role = "user", content = text("Q2") }, + } + assert_order(claude_order(claude_mapper.map_messages(rows).messages[2]), { "kept" }) + assert_order(bedrock_order(bedrock_mapper.map_messages(rows).messages[2]), { "kept" }) + end) + + it("keeps outer whitespace of visible text", function() + local rows = { + { role = "user", content = text("Q") }, + { role = "assistant", content = text(" kept\n") }, + { role = "user", content = text("Q2") }, + } + assert_order(claude_order(claude_mapper.map_messages(rows).messages[2]), { " kept\n" }) + assert_order(bedrock_order(bedrock_mapper.map_messages(rows).messages[2]), { " kept\n" }) + end) + + it("moves a cache marker off a blank turn", function() + local mapped = claude_mapper.map_messages({ + { role = "user", content = text("Q") }, + { role = "assistant", content = text(" ") }, + { role = "cache_marker", marker_id = "tail" }, + }) + test.eq(#mapped.messages, 1) + test.not_nil(mapped.messages[1].content[1].cache_control) + end) + end) + + describe("effective output", function() + local function setup(handler, method, card, policy) + llm._usage_tracker = false + use_card(card) + llm._providers = policy_providers(handler, method, policy or BUDGET_POLICY) + end + + local function card(fields) + local out = { name = "m", id = "m", capabilities = {}, class = {}, + providers = { { id = "route", provider_model = "wire" } } } + for key, value in pairs(fields) do out[key] = value end + return out + end + + it("clamps the Claude budget raise to the card limit and reports the wire value", function() + local sent = capture_claude(claude_generate) + setup(claude_generate, "generate", card({ output_tokens = 8192 })) + local result: any = assert(llm.generate("hi", { model = "m", max_tokens = 512, thinking_effort = 100 })) + test.eq(sent[1].payload.max_tokens, 8192) + test.eq(sent[1].payload.thinking.budget_tokens, 7168) + test.eq(result.metadata.adjusted.max_tokens.requested, 512) + test.eq(result.metadata.adjusted.max_tokens.sent, 8192) + test.eq(result.metadata.adjusted.thinking_budget.requested, 24000) + test.eq(result.metadata.adjusted.thinking_budget.sent, 7168) + end) + + it("clamps the Bedrock budget raise to the card limit and reports the wire value", function() + local sent = capture_bedrock(bedrock_generate) + setup(bedrock_generate, "generate", card({ output_tokens = 8192 })) + local result: any = assert(llm.generate("hi", { model = "m", max_tokens = 512, thinking_effort = 100 })) + test.eq(sent[1].payload.inferenceConfig.maxTokens, 8192) + test.eq(sent[1].payload.additionalModelRequestFields.thinking.budget_tokens, 7168) + test.eq(result.metadata.adjusted.max_tokens.sent, 8192) + end) + + it("reserves Claude answer room when the capped output just exceeds the budget", function() + local sent = capture_claude(claude_generate) + setup(claude_generate, "generate", card({ output_tokens = 8192 })) + local result: any = assert(llm.generate("hi", { model = "m", max_tokens = 512, thinking_effort = 31 })) + test.eq(sent[1].payload.max_tokens, 8192) + test.eq(sent[1].payload.thinking.budget_tokens, 7168) + test.eq(result.metadata.adjusted.max_tokens.requested, 512) + test.eq(result.metadata.adjusted.max_tokens.sent, 8192) + test.eq(result.metadata.adjusted.thinking_budget.requested, 8147) + test.eq(result.metadata.adjusted.thinking_budget.sent, 7168) + end) + + it("reserves Bedrock answer room when the capped output just exceeds the budget", function() + local sent = capture_bedrock(bedrock_generate) + setup(bedrock_generate, "generate", card({ output_tokens = 8192 })) + local result: any = assert(llm.generate("hi", { model = "m", max_tokens = 512, thinking_effort = 31 })) + test.eq(sent[1].payload.inferenceConfig.maxTokens, 8192) + test.eq(sent[1].payload.additionalModelRequestFields.thinking.budget_tokens, 7168) + test.eq(result.metadata.adjusted.max_tokens.requested, 512) + test.eq(result.metadata.adjusted.max_tokens.sent, 8192) + test.eq(result.metadata.adjusted.thinking_budget.requested, 8147) + test.eq(result.metadata.adjusted.thinking_budget.sent, 7168) + end) + + it("reports a raise with no limit in the way", function() + local sent = capture_claude(claude_generate) + setup(claude_generate, "generate", card({})) + local result: any = assert(llm.generate("hi", { model = "m", max_tokens = 512, thinking_effort = 100 })) + test.eq(sent[1].payload.max_tokens, 25024) + test.eq(sent[1].payload.thinking.budget_tokens, 24000) + test.eq(result.metadata.adjusted.max_tokens.requested, 512) + test.eq(result.metadata.adjusted.max_tokens.sent, 25024) + test.is_nil(result.metadata.adjusted.thinking_budget) + end) + + it("leaves a request above the budget and under the limit alone", function() + local sent = capture_claude(claude_generate) + setup(claude_generate, "generate", card({ output_tokens = 64000 })) + local result: any = assert(llm.generate("hi", { model = "m", max_tokens = 30000, thinking_effort = 100 })) + test.eq(sent[1].payload.max_tokens, 30000) + test.eq(sent[1].payload.thinking.budget_tokens, 24000) + test.is_nil(result.metadata.adjusted) + end) + + it("rejects the raise under strict", function() + capture_claude(claude_generate) + setup(claude_generate, "generate", card({})) + local result, err = llm.generate("hi", { model = "m", max_tokens = 512, thinking_effort = 100, strict = true }) + test.is_nil(result) + test.contains(tostring(err), "max_tokens") + end) + + it("gives up thinking when the limit leaves no room for the minimum budget", function() + local sent = capture_claude(claude_generate) + setup(claude_generate, "generate", card({ output_tokens = 1000 })) + local result: any = assert(llm.generate("hi", { model = "m", max_tokens = 512, thinking_effort = 100 })) + test.is_nil(sent[1].payload.thinking) + test.eq(sent[1].payload.max_tokens, 512) + test.eq(result.metadata.adjusted.thinking_effort.requested, 100) + end) + + it("admits on the raised output", function() + local sent = capture_claude(claude_generate) + setup(claude_generate, "generate", card({ max_tokens = 30000 })) + local result, err = llm.generate(string.rep("x", 80000), { model = "m", max_tokens = 512, thinking_effort = 100 }) + test.is_nil(result) + test.contains(tostring(err), "context window") + test.eq(#sent, 0) + end) + + it("reserves the driver default when max_tokens is unset", function() + local sent = capture_claude(claude_generate) + setup(claude_generate, "generate", card({ max_tokens = 1000 })) + local result, err = llm.generate(string.rep("x", 800), { model = "m" }) + test.is_nil(result) + test.contains(tostring(err), "context window") + test.eq(#sent, 0) + end) + + it("keeps the wire default for an unset max_tokens under the limit without adjustment", function() + local sent = capture_bedrock(bedrock_generate) + setup(bedrock_generate, "generate", card({ output_tokens = 64000 })) + local result: any = assert(llm.generate("hi", { model = "m" })) + test.eq(sent[1].payload.inferenceConfig.maxTokens, 2000) + test.is_nil(result.metadata.adjusted) + end) + + it("clamps the driver default to the card limit", function() + local sent = capture_claude(claude_generate) + setup(claude_generate, "generate", card({ output_tokens = 1000 })) + local result: any = assert(llm.generate("hi", { model = "m" })) + test.eq(sent[1].payload.max_tokens, 1000) + test.is_nil(result.metadata.adjusted) + end) + + it("applies to structured output", function() + local sent = capture_claude(claude_structured) + setup(claude_structured, "structured_output", card({ output_tokens = 8192 })) + local result: any = assert(llm.structured_output(SCHEMA, "hi", { model = "m", max_tokens = 512, thinking_effort = 100 })) + test.eq(sent[1].payload.max_tokens, 8192) + test.eq(result.metadata.adjusted.max_tokens.sent, 8192) + end) + + it("leaves a driver without a policy to its request", function() + local sent = capture_openai(openai_generate) + setup(openai_generate, "generate", card({ max_tokens = 1000 }), false) + llm._providers.driver_output_policy = function() return nil, nil end + assert(llm.generate(string.rep("x", 800), { model = "m" })) + test.is_nil(sent[1].payload.max_output_tokens) + end) + + it("declares the policy on the real driver bindings", function() + for _, driver in ipairs({ + { id = "wippy.llm.claude:driver", mapper = claude_mapper }, + { id = "wippy.llm.bedrock:driver", mapper = bedrock_mapper }, + }) do + local entry = assert(registry.get(driver.id)) + test.eq(entry.meta.output_policy.thinking, driver.mapper.CAPABILITY.defaults.thinking) + test.eq(entry.meta.output_policy.default_max_tokens, driver.mapper.DEFAULT_MAX_TOKENS) + end + test.is_nil(assert(registry.get("wippy.llm.openai:driver")).meta.output_policy) + end) + + it("reads the policy through the provider discovery", function() + local entry = assert(registry.get("wippy.llm.bedrock:provider")) + local policy, err = providers.driver_output_policy(entry.id) + test.is_nil(err) + test.eq(policy.thinking, "budget") + test.eq(policy.default_max_tokens, 2000) + end) + end) + + describe("blank user, system and developer text", function() + local function rows() + return { + { role = "system", content = text("") }, + { role = "system", content = " \n" }, + { role = "system", content = text("S") }, + { role = "user", content = { { type = "text", text = " " }, { type = "text", text = "Q" } } }, + { role = "user", content = text("\t") }, + { role = "developer", content = text(" ") }, + { role = "developer", content = text(""), metadata = { preserve_position = true } }, + { role = "system", content = text(" "), metadata = { preserve_position = true } }, + } + end + + it("never reaches Claude", function() + local mapped = claude_mapper.map_messages(rows()) + test.eq(#mapped.system, 1) + test.eq(mapped.system[1].text, "S") + test.eq(#mapped.messages, 1) + test.eq(#mapped.messages[1].content, 1) + test.eq(mapped.messages[1].content[1].text, "Q") + end) + + it("never reaches Bedrock", function() + local mapped = bedrock_mapper.map_messages(rows()) + test.eq(#mapped.system, 1) + test.eq(#mapped.messages, 1) + test.eq(#mapped.messages[1].content, 1) + end) + + it("never reaches OpenAI", function() + local input = rows() + test.eq(openai_mapper.extract_instructions(input), "S") + local items = openai_mapper.map_messages(input, {}) + test.eq(#items, 1) + test.eq(#items[1].content, 1) + end) + + it("never reaches OpenAI-compatible", function() + local mapped = compat_mapper.map_messages(rows(), {}) + test.eq(#mapped, 2) + test.eq(mapped[1].role, "system") + test.eq(mapped[2].role, "user") + test.eq(#mapped[2].content, 1) + end) + + it("never reaches OpenAI-compatible structured output", function() + local sent = capture_openai(compat_structured) + compat_structured.handler({ model = "m", messages = rows(), schema = SCHEMA, options = {} }) + test.eq(#sent[1].payload.messages, 2) + end) + + it("never reaches Google", function() + local mapped, system = google_mapper.map_messages(rows(), {}) + test.eq(#system, 1) + test.eq(#mapped, 1) + test.eq(#mapped[1].parts, 1) + end) + + it("sends no blank part in place of a part OpenAI cannot render", function() + local document = { type = "document", source = { type = "base64", mime_type = "application/pdf", data = "BBBB" } } + local items = openai_mapper.map_messages({ + { role = "user", content = { { type = "text", text = "See" }, document } }, + { role = "user", content = { document } }, + }, {}) + test.eq(#items, 1) + test.eq(#items[1].content, 1) + end) + + it("keeps an image beside blank text", function() + local image = { type = "image", source = { type = "base64", mime_type = "image/png", data = "AAAA" } } + local mapped = claude_mapper.map_messages({ { role = "user", content = { { type = "text", text = " " }, image } } }) + test.eq(#mapped.messages[1].content, 1) + test.eq(mapped.messages[1].content[1].type, "image") + end) + + it("keeps an empty tool result", function() + local mapped = claude_mapper.map_messages({ + { role = "function_call", content = {}, function_call = { id = "c1", name = "t", arguments = "{}" } }, + { role = "function_result", name = "t", function_call_id = "c1", content = text("") }, + }) + test.eq(mapped.messages[2].content[1].type, "tool_result") + end) + + it("leaves the caller's rows untouched", function() + local input = rows() + claude_mapper.map_messages(input) + test.eq(#input, 8) + test.eq(#input[4].content, 2) + end) + end) + + describe("requests with no turn", function() + local SYSTEM_ONLY = { { role = "system", content = text("S") } } + local BLANK_USER = { { role = "user", content = text(" ") } } + local CASES = { + { name = "Claude generate", handler = claude_generate, capture = capture_claude }, + { name = "Claude structured", handler = claude_structured, capture = capture_claude, schema = true }, + { name = "Claude native structured", handler = claude_structured, capture = capture_claude, schema = true, accepts = { structured_output = "native" } }, + { name = "Bedrock generate", handler = bedrock_generate, capture = capture_bedrock }, + { name = "Bedrock structured", handler = bedrock_structured, capture = capture_bedrock, schema = true }, + { name = "Google generate", handler = google_generate, capture = capture_google }, + { name = "Google structured", handler = google_structured, capture = capture_google, schema = true }, + { name = "OpenAI generate", handler = openai_generate, capture = capture_openai }, + { name = "OpenAI structured", handler = openai_structured, capture = capture_openai, schema = true }, + } + for _, case in ipairs(CASES) do + for label, messages in pairs({ ["system only"] = SYSTEM_ONLY, ["blank user"] = BLANK_USER }) do + it("rejects before HTTP on " .. case.name .. ": " .. label, function() + local sent = case.capture(case.handler) + local response, err = case.handler.handler({ model = "m", messages = messages, options = {}, + accepts = case.accepts, schema = case.schema and SCHEMA or nil }) + test.is_nil(response) + test.eq(#sent, 0) + test.eq(err:kind(), "Invalid") + test.contains(err:message(), "no conversation turn") + end) + end + end + + it("still sends a system-role note alone on OpenAI", function() + local sent = capture_openai(openai_generate) + assert(openai_generate.handler({ model = "m", options = {}, messages = { + { role = "system", content = text("N"), metadata = { preserve_position = true } } } })) + test.eq(#sent, 1) + test.eq(#sent[1].payload.input, 1) + end) + + it("still sends a system message alone on OpenAI-compatible", function() + local sent = capture_openai(compat_generate) + assert(compat_generate.handler({ model = "m", options = {}, messages = SYSTEM_ONLY })) + test.eq(#sent, 1) + test.eq(#sent[1].payload.messages, 1) + end) + end) + + describe("drop_stale_thinking on string content", function() + it("handles an assistant message with string content", function() + local builder: any = prompt.new({ + { role = "user", content = "q" }, + { role = "assistant", content = "hi", metadata = { thinking_blocks = { signed("A") } } }, + { role = "user", content = "q2" }, + } :: any) + local ok, result, err = pcall(function() return builder:drop_stale_thinking(function() return false end) end) + test.is_true(ok) + test.is_nil(err) + test.is_nil(result.messages[2].metadata.thinking_blocks) + end) + end) + + describe("foreign tool call ids", function() + local LONG = "get_customer_subscription_details_0194c7a2-9c1e-7b3f-8a4d-1234567890ab" + local LONG_SIBLING = "get_customer_subscription_details_0194c7a2-9c1e-7b3f-8a4d-1234567890ac" + + local function round(id): any + return { + { role = "user", content = text("q") }, + { role = "function_call", content = {}, function_call = { id = id, name = "t", arguments = "{}" } }, + { role = "function_result", name = "t", function_call_id = id, content = text("r") }, + } + end + + local function bedrock_ids(rows): any + local messages: any = bedrock_mapper.map_messages(rows).messages + local ids = {} + for _, message in ipairs(messages) do + for _, block in ipairs(message.content) do + if block.toolUse then table.insert(ids, block.toolUse.toolUseId) end + if block.toolResult then table.insert(ids, block.toolResult.toolUseId) end + end + end + return ids + end + + local function openai_ids(rows): any + local ids = {} + for _, item in ipairs(openai_mapper.map_messages(rows, {})) do + if item.call_id then table.insert(ids, item.call_id) end + end + return ids + end + + it("fits the same 70-character id on a Bedrock call and result", function() + test.eq(#LONG, 70) + local ids = bedrock_ids(round(LONG)) + test.eq(#ids, 2) + test.eq(ids[1], ids[2]) + test.is_true(#ids[1] <= 64) + test.not_nil(ids[1]:match("^[a-zA-Z0-9_-]+$")) + end) + + it("translates ids with characters Bedrock rejects", function() + local ids = bedrock_ids(round("provider:call/one")) + test.eq(ids[1], ids[2]) + test.not_nil(ids[1]:match("^[a-zA-Z0-9_-]+$")) + end) + + it("keeps Bedrock ids stable over builds", function() + test.eq(bedrock_ids(round(LONG))[1], bedrock_ids(round(LONG))[1]) + end) + + it("keeps two long Bedrock ids sharing a prefix apart", function() + local rows = round(LONG) + for _, row in ipairs(round(LONG_SIBLING)) do + if row.role ~= "user" then table.insert(rows, row) end + end + local ids = bedrock_ids(rows) + test.eq(#ids, 4) + test.eq(ids[1], ids[2]) + test.eq(ids[3], ids[4]) + test.is_false(ids[1] == ids[3]) + end) + + it("avoids a Bedrock native id equal to a translated one", function() + local translated = bedrock_ids(round(LONG))[1] + local rows = round(LONG) + for _, row in ipairs(round(translated)) do + if row.role ~= "user" then table.insert(rows, row) end + end + local ids = bedrock_ids(rows) + test.is_false(ids[1] == ids[3]) + test.eq(ids[3], translated) + test.is_true(#ids[1] <= 64) + end) + + it("leaves ordinary ids byte-identical on Bedrock and OpenAI", function() + local ordinary = "tooluse_Abc-123_" .. string.rep("x", 48) + test.eq(#ordinary, 64) + test.eq(bedrock_ids(round(ordinary))[1], ordinary) + test.eq(bedrock_ids(round(ordinary))[2], ordinary) + test.eq(openai_ids(round(ordinary))[1], ordinary) + test.eq(openai_ids(round("call:with/other-chars"))[1], "call:with/other-chars") + end) + + it("translates a function_call content part on Bedrock", function() + local messages: any = bedrock_mapper.map_messages({ + { role = "assistant", content = { { type = "function_call", id = LONG, name = "t", arguments = {} } } }, + { role = "function_result", name = "t", function_call_id = LONG, content = text("r") }, + }).messages + local call = messages[1].content[1].toolUse.toolUseId + test.is_true(#call <= 64) + test.eq(messages[2].content[1].toolResult.toolUseId, call) + end) + + it("fits the same long id on an OpenAI call and output", function() + local ids = openai_ids(round(LONG)) + test.eq(#ids, 2) + test.eq(ids[1], ids[2]) + test.is_true(#ids[1] <= 64) + test.eq(openai_ids(round(LONG))[1], ids[1]) + end) + + it("keeps the layout of a Bedrock response with a long id", function() + local result = bedrock_mapper.format_success_response({ output = { message = { role = "assistant", content = { + reasoning_block("s1"), { text = "a" }, reasoning_block("s2"), + { toolUse = { toolUseId = LONG, name = "t", input = { a = 1 } } }, + } } }, stopReason = "tool_use", usage = { inputTokens = 1, outputTokens = 1 } }, {}) + local order = bedrock_order(bedrock_replay(stored_rows(result)).messages[2]) + test.eq(order[3], "s2") + test.is_true(#order[4] <= 64) + end) + + it("keeps Claude's translated ids as they are", function() + local messages: any = claude_mapper.map_messages(round("provider:call/one")).messages + test.eq(#messages[2].content[1].id, 69) + test.eq(messages[2].content[1].id:sub(1, 5), "tool-") + end) + end) + + describe("discard_calls", function() + it("removes calls and the thinking behind the last visible text", function() + local metadata: any = { + thinking_blocks = { { type = "thinking", signature = "s1", position = 1 }, + { type = "thinking", signature = "s2", position = 3 } }, + content_positions = { { type = "text", position = 2, text = "A" }, + { type = "tool_use", position = 4, id = "c1" } }, + } + thinking.discard_calls(metadata, "A") + test.eq(#metadata.thinking_blocks, 1) + test.eq(metadata.thinking_blocks[1].signature, "s1") + test.eq(#metadata.content_positions, 1) + test.eq(metadata.content_positions[1].type, "text") + end) + + it("clears thinking of a plain-order response with no visible text", function() + local metadata: any = { thinking_blocks = { signed("s1") } } + thinking.discard_calls(metadata, " ") + test.eq(#metadata.thinking_blocks, 0) + end) + + it("keeps thinking of a plain-order response with visible text", function() + local metadata: any = { thinking_blocks = { signed("s1") } } + thinking.discard_calls(metadata, "A") + test.eq(#metadata.thinking_blocks, 1) + end) + + it("leaves no layout when no visible text remains", function() + local metadata: any = { + thinking_blocks = { { type = "thinking", signature = "s1", position = 2 } }, + content_positions = { { type = "text", position = 1, text = " " }, + { type = "tool_use", position = 3, id = "c1" } }, + } + thinking.discard_calls(metadata, " ") + test.eq(#metadata.thinking_blocks, 0) + test.is_nil(metadata.content_positions) + end) + end) + + describe("mark_stale_thinking", function() + it("marks every block carrier up to the last stale one on copies", function() + local first = { role = "assistant", content = text("a"), metadata = { thinking_blocks = { signed("A") }, epoch = "old" } } + local second = { role = "assistant", content = text("b"), metadata = { thinking_blocks = { signed("B") }, epoch = "new" } } + local third = { role = "assistant", content = text("c"), metadata = { thinking_blocks = { signed("C") }, epoch = "new" } } + local builder: any = prompt.new({ { role = "user", content = text("q") }, second, first, third } :: any) + builder:mark_stale_thinking(function(msg) return msg.metadata.epoch == "new" end) + test.is_true(builder.messages[2].metadata.stale_thinking) + test.is_true(builder.messages[3].metadata.stale_thinking) + test.is_nil(builder.messages[4].metadata.stale_thinking) + test.is_nil(first.metadata.stale_thinking) + test.is_nil(second.metadata.stale_thinking) + end) + + it("handles rows without blocks and string content", function() + local builder: any = prompt.new({ + { role = "user", content = "q" }, + { role = "assistant", content = "hi" }, + } :: any) + builder:mark_stale_thinking(function() return false end) + test.is_nil(builder.messages[2].metadata) + end) + end) + end) +end + +return test.run_cases(define_tests) diff --git a/src/llm/test/route_flow_test.lua b/src/llm/test/route_flow_test.lua index d2b05d5..1161544 100644 --- a/src/llm/test/route_flow_test.lua +++ b/src/llm/test/route_flow_test.lua @@ -78,7 +78,7 @@ local function define_tests() } end handler._client = { ENDPOINTS = { MESSAGES = "/messages" }, request = request, converse = request } - llm._providers = { driver_declares_legacy_reasoning_flag = function() return false end, open = function() + llm._providers = { driver_declares_legacy_reasoning_flag = function() return false end, driver_output_policy = function() return nil, nil end, open = function() return { [method] = function(_, args) return handler.handler(args) end } end } return sent @@ -194,7 +194,7 @@ local function define_tests() } end handler._client = { ENDPOINTS = { MESSAGES = "/messages" }, request = request, converse = request } - llm._providers = { driver_declares_legacy_reasoning_flag = function() return false end, open = function() + llm._providers = { driver_declares_legacy_reasoning_flag = function() return false end, driver_output_policy = function() return nil, nil end, open = function() return { generate = function(_, args) return handler.handler(args) end } end } return sent @@ -363,7 +363,7 @@ local function define_tests() for key, value in pairs(driver.accepts) do ref[key] = value end local card = { name = "route-model", providers = { ref } } llm._model_resolver = { resolve = function() return card end } - llm._providers = { driver_declares_legacy_reasoning_flag = function() return false end, open = function() + llm._providers = { driver_declares_legacy_reasoning_flag = function() return false end, driver_output_policy = function() return nil, nil end, open = function() return { [method] = function(_, args) return handler.handler(args) end } end } return sent @@ -460,7 +460,7 @@ local function define_tests() options = { thinking_effort = effort, request = request } } } } llm._model_resolver = { resolve = function() return card end } - llm._providers = { driver_declares_legacy_reasoning_flag = function() return false end, open = function() + llm._providers = { driver_declares_legacy_reasoning_flag = function() return false end, driver_output_policy = function() return nil, nil end, open = function() return { [method] = function(_, args) return handler.handler(args) end } end } local _, err @@ -530,7 +530,7 @@ local function define_tests() for key, value in pairs(driver.accepts) do ref[key] = value end local card = { name = "route-model", providers = { ref } } llm._model_resolver = { resolve = function() return card end } - llm._providers = { driver_declares_legacy_reasoning_flag = function() return false end, open = function() + llm._providers = { driver_declares_legacy_reasoning_flag = function() return false end, driver_output_policy = function() return nil, nil end, open = function() return { structured_output = function(_, args) return handler.handler(args) end } end } local result, err = call("structured_output", { model = "route-model" }) @@ -550,7 +550,7 @@ local function define_tests() local card = { name = "route-model", providers = { { id = "route", provider_model = "wire", thinking = "adaptive", options = { request = { usage = { include = false } } } } } } llm._model_resolver = { resolve = function() return card end } - llm._providers = { driver_declares_legacy_reasoning_flag = function() return false end, open = function() + llm._providers = { driver_declares_legacy_reasoning_flag = function() return false end, driver_output_policy = function() return nil, nil end, open = function() return { generate = function(_, args) return openai_compat_generate.handler(args) end } end } local result, err = call("generate", { model = "route-model" }) @@ -562,7 +562,7 @@ local function define_tests() for _, method in ipairs({ "generate", "structured_output" }) do it("returns an invalid request from Google " .. method .. " for declared thinking", function() local handler = method == "generate" and google_generate or google_structured - llm._providers = { driver_declares_legacy_reasoning_flag = function() return false end, open = function() + llm._providers = { driver_declares_legacy_reasoning_flag = function() return false end, driver_output_policy = function() return nil, nil end, open = function() return { [method] = function(_, args) return handler.handler(args) end } end } local result, err = call(method, { provider_id = "route", model = "wire", accepts = { thinking = "adaptive" } }) diff --git a/src/llm/test/util/route_test.lua b/src/llm/test/util/route_test.lua index c164606..646eaf1 100644 --- a/src/llm/test/util/route_test.lua +++ b/src/llm/test/util/route_test.lua @@ -112,6 +112,38 @@ local function define_tests() end) end) + describe("Route output planning", function() + it("reserves answer room when a cap falls just above the thinking budget", function() + for _, case in ipairs({ + { limit = 8192, output = 8192, budget = 7168 }, + { limit = 8148, output = 8148, budget = 7124 }, + { limit = 9170, output = 9170, budget = 8146 }, + { limit = 9171, output = 9171, budget = 8147 }, + { limit = 10000, output = 9171, budget = 8147 }, + }) do + local plan = route.plan_output(512, 31, "budget", 2000, case.limit) + test.eq(plan.max_tokens, case.output) + test.eq(plan.budget, case.budget) + test.eq(plan.wanted_budget, 8147) + test.is_nil(plan.dropped_thinking) + end + equal(route.plan_output(8192, 31, "budget", 2000, nil), + { max_tokens = 8192, budget = 8147, wanted_budget = 8147 }) + end) + + it("keeps the minimum thinking budget when a cap leaves less than the answer room", function() + for _, limit in ipairs({ 1255, 2047, 2048, 1025 }) do + local plan = route.plan_output(512, 1, "budget", 2000, limit) + test.eq(plan.max_tokens, limit) + test.eq(plan.budget, 1024) + test.eq(plan.wanted_budget, 1254) + test.is_nil(plan.dropped_thinking) + end + equal(route.plan_output(512, 1, "budget", 2000, 1024), + { max_tokens = 512, dropped_thinking = true }) + end) + end) + describe("Route capability helpers", function() local capability = { name = "Test",