Repository navigation
fix(openai-compatible): make strict mode opt-in for structured outputs - #446
Merged
Merged
Conversation
OpenAICompletionsProvider unconditionally sent strict: true in structured-output response_format bodies. Strict mode (constrained decoding) is an OpenAI implementation detail, not part of JSON Schema, and is only safe when the backend actually enforces it -- several OpenAI-compatible backends reject the flag or mishandle the schema. Strict mode is now gated on a provider flag (default off): - ChatGroq() and ChatAzureOpenAICompletions() opt in (both enforce it). - ChatPosit() opts in for models whose id starts with 'openai/'. - ChatOpenAICompletions() auto-detects: strict when base_url points at api.openai.com, standard JSON Schema otherwise; its new strict argument overrides the heuristic. - ChatOpenRouter(), ChatPortkey(), and ChatHuggingFace() gain a strict argument (default False) for endpoints where the right choice can't be determined automatically. Recorded request bodies in the five affected provider cassettes were hand-updated to drop the flag; recorded responses are untouched. Ports tidyverse/ellmer#1135.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Ports tidyverse/ellmer#1135.
Problem
OpenAICompletionsProvider— the base for every OpenAI-compatible provider — unconditionally sentstrict: truein structured-outputresponse_formatbodies. Strict mode (constrained decoding) is an OpenAI implementation detail, not part of JSON Schema, and it's only safe when the backend actually enforces it. Backends that do prompt-based structured output read the schema the standard way, and some reject unknown flags outright.Note chatlas never had the other two leaks ellmer's PR fixed: tool definitions here already omit
strictunless the user opts in per-tool (Tool(strict=True)), and schemas are standard pydantic JSON Schema (no all-required-plus-nullable convention). So this port is scoped to theresponse_formatflag.Changes
OpenAICompletionsProvidergains astrictflag (defaultFalse) that gatesstrict: truein theresponse_formatbody:ChatGroq(),ChatAzureOpenAICompletions(), andChatPosit()models whose id starts withopenai/(kept in sync acrossset_model()).ChatOpenAICompletions()— chatlas's dual-duty entry point for both OpenAI-proper and third-party backends — enables strict mode whenbase_urlpoints atapi.openai.comand disables it otherwise. No behavior change for existing OpenAI users; third-party backends (vLLM, Ollama, LiteLLM, ...) get standard JSON Schema by default. A newstrictargument overrides the heuristic.strictargument (defaultFalse):ChatOpenRouter(),ChatPortkey(),ChatHuggingFace(), for endpoints where the right choice can't be determined automatically.ChatDatabricks(),ChatMistral(),ChatDeepSeek(),ChatPerplexity(),ChatCloudflare(),ChatOllama(),ChatLMStudio(),ChatGithub()— these no longer sendstrict: true.ChatGithub()is the one judgment call: ellmer's PR doesn't mentionchat_github(), so it stays opted out here.Cassettes
Recorded request bodies in the five affected cassettes (openrouter, cloudflare, databricks, mistral, huggingface data-extraction tests) were hand-updated to drop the flag; recorded responses are untouched.
Tests
New credential-free tests cover: the
response_formatgating on both branches,ChatOpenAICompletions()base-url auto-detection and explicit override, per-provider defaults (groq/openrouter/portkey/huggingface/azure), and theChatPosit()openai/heuristic includingset_model()switches.