[api][integrations][java][python] Ask each connection whether it could apply a schema natively

[weiqingy]

Linked issue: #912

Builds on #1128, now merged, which added the effective-model hook this uses. Rebased onto it, so the diff here is this change alone.

Purpose of change

Nothing changes for a caller yet: no production code asks what this adds. It is the second step of wiring an agent output_schema through to the providers that can enforce it, and it removes a drift hazard the first step exposed.

Whether a native schema reaches a provider depends on more than the model. The schema form decides it, and so do conditions that differ per provider: an api version floor on Azure, a caller supplied output config on Anthropic, bound tools on Gemini. Each connection already made that decision inside its own request path, where a caller cannot see it. Asking a connection to answer the same question separately would mean writing those conditions twice, and two copies drift silently, because a caller only ever sees the answer.

Runtime flow

Both connection bases gain a feasibility query: can this connection apply this schema, to a request built from these tools and parameters, with the effective model's capability set aside. The default is false, which suits a connection that translates no schema at all.

Every connection with a native branch implements it, and each branch now calls the query instead of restating its conditions, so the branch and the answer cannot disagree. Capability stays a separate conjunct of each branch. The query is asked with the parameters the caller supplied, before a request path strips its own keys, because two connections read keys that a stripped copy no longer holds.

Key decisions

Feasibility is asked separately from capability, and neither bounds the other. A schema form a connection cannot translate is infeasible on a model it calls capable; a translatable form is feasible on a model it calls incapable. Merging them would have hidden one behind the other.

Each override answers from its own request path rather than from a copy of its conditions. That is what the tests pin, and it is why the query is asked with the unstripped parameters.

Anthropic's JSON prefill keeps reading the live parameters rather than sharing the query's snapshot, because prefill has to see the output config the branch derives, which the caller never supplied.

Behavioral Semantics

Interaction decisions

Connection	Conditions the query answers	Kept in the branch
OpenAI, Ollama, Bedrock, watsonx, Tongyi	schema form	capability
Azure	schema form, api version floor	capability
Anthropic	schema form, no caller supplied output config	capability
Gemini	schema form, no bound tools	capability

Behavioral contracts

1. The query answers whether the connection could apply that schema to such a request, with the effective model's capability excluded.
2. It accepts a missing schema, missing tools and missing parameters without raising, so a caller can ask speculatively, and it reads the parameters without consuming them.
3. A false answer is not an error: the caller keeps the prompt fallback.
4. A true answer is not a promise the call succeeds. A connection may still raise once its branch has decided to apply, where the caller supplied a conflicting format, or where the schema cannot be rendered.
5. An override answers from the same logic its own request path uses.

Failure behavior

No new failure path, and one narrow change on invalid input. A branch consults the query before the capability predicate, so a request the query reports infeasible no longer evaluates that predicate. Where the model is not a string, three Python connections raised there and now send the request without a schema: OpenAI at its modality check, Anthropic at its alias scan, and vLLM, which inherits the query and classifies by trimming the name. Tongyi and Azure already answered False quietly, since both match by set membership. Anthropic reaches that path with a translatable schema when the caller supplied an output config, and Azure when the api version is below its floor, since both conditions moved into the query. Ollama and watsonx are unaffected. Measured by driving every connection across 1032 request shapes on both trees: 70 cells differ, none of them carrying a valid or absent model, and all 696 that do are byte-identical.

Tests

Contract	Tests
1, 2, 3	the base-class tests in both languages: default, missing inputs accepted, parameters not consumed
4	each connection's contract states it; the conflicting-format raises are pinned by existing tests
5	a binding test per connection, capturing what the branch actually does and comparing it with the answer
the exclusion	a dedicated test per connection that capability is not folded into the answer
the unstripped parameters	a test per stripping connection that the query sees keys the request path later removes

Coverage by risk. The failure that matters is an answer that stops matching the branch, which no assertion against a literal can catch. The binding tests capture the model or the request the branch produced and compare it with the query, so a branch that stops agreeing fails. The exclusion tests exist because a binding test is structurally blind to capability being folded into an override: it moves both sides together, which was measured on both languages.

Suite results: Java api 428, chat-models common 6, openai 131, anthropic 103, bedrock 90, gemini 71, ollama 18, watsonx 51. Python 797 passed, 12 skipped.

Not verified. No live provider was called.

API

New method on each chat-model connection base, protected in Java and public in Python, plus overrides. No user-facing API changes, and no behavior change for an existing caller beyond the invalid-model case above.

Documentation

• doc-needed
• doc-not-needed
• doc-included

Was this patch authored or co-authored using generative AI tooling?

• Yes
• No

Generated-by: Claude Code 2.1.272 (Claude Opus 5)

[wenjin272]

Thanks for the thorough Java/Python implementation and tests. I left two API-design comments: one on consolidating the connection-level predicates, and one non-blocking follow-up for flat RowTypeInfo native support.

[wenjin272]

Do we need two connection-level predicates here? Although model capability and request/schema feasibility are independent dimensions internally, the framework ultimately needs one answer: whether native structured output can be applied to this request. Could canApplyNativeStructuredOutput resolve the effective model internally and incorporate capability, replacing supportsNativeStructuredOutput as a connection-level hook? Providers could still keep model-specific allowlists as private helpers. I do not see a production caller that needs to consume the two dimensions independently, so exposing both seems to add API and composition complexity without a caller requirement.

[weiqingy]

You're right that nothing consumes the two independently today, and the combining happens inside each connection rather than in the framework: canApplyNativeStructuredOutput(...) && supportsNativeStructuredOutput(...) at OpenAICompletionsConnection.java:303-304, and the same pair in the Anthropic, Bedrock, Gemini, Ollama, Azure and watsonx connections.

The caller that would read capability on its own is StructuredOutputStrategy.resolvesToNative (StructuredOutputStrategy.java:65-75), and it is not wired to anything yet. The setup parses and stores the strategy without reading it (BaseChatModelSetup.java:71-74), and it still calls the three-argument chat (:200), so no schema reaches a connection through the framework at all on this branch. The independence is a designed contract here rather than observed behavior.

I would still keep them separate, and the reason that already shows up in code is provider inheritance. VLLMChatModelConnection extends OpenAICompletionsConnection and overrides capability only (VLLMChatModelConnection.java:80-83), because the inherited allowlist rejects served models such as Qwen/Qwen2.5-7B-Instruct, while it inherits feasibility unchanged. Merge the two and super stops being usable, since the parent's answer has already applied the allowlist the subclass exists to escape. vLLM would have to restate the parent's feasibility rule locally: one line in Java, and in Python also a reach into a module-private helper in the parent module (vllm_chat_model.py:77-85).

Your point holds for the other providers though. Ollama and watsonx answer capability with an unconditional true (OllamaChatModelConnection.java:203, WatsonxChatModelConnection.java:265), so under a merged predicate their override collapses into the feasibility test and disappears.

The contract does state that the two are independent (BaseChatModelConnection.java:106-112), but not why they are separate hooks rather than one, which is a fair gap. I will state the asymmetry at the declaration: capability is advisory, so a policy is allowed to override it, while feasibility bounds what the connection can encode at all.

If you would rather have one hook, the shape that keeps both behaviors is a three-valued answer rather than a boolean, so that "cannot encode this" and "this model probably will not honor it" stay distinguishable at a single call site. That is more API surface than the split rather than less, which is why I have not taken it, but it is a real option if the single call site is worth it.

[weiqingy]

Pushed as 7f6417d7. Both declarations now state the asymmetry in each language: capability is advisory and a configured policy may overrule it in either direction, while feasibility is binding, since a request whose schema the connection cannot encode has no native form to send.

Writing it turned up a related error in four places, all claiming that policy and capability are combined at request-build time. That is wrong on both halves: policy is parsed and stored but combined nowhere yet, and what a connection combines while building a request is capability with feasibility. Each site now names resolvesToNative as where policy meets capability, which is a declared method rather than a call path, so it stays accurate while nothing calls it.

One of the four is a pydantic field description rather than a comment, so that one change is a runtime string. Nothing reads it today.

[wenjin272]

Following up on my earlier concern: I agree that effective-model resolution, model capability, and request/schema feasibility are all necessary concepts. My concern is not about removing those checks, but about exposing all three as independent hooks on BaseChatModelConnection. That makes the internal steps of one structured-output decision part of every connection’s extension contract.

Comparing this with LangChain and AgentScope Java suggests a simpler boundary. LangChain uses a request-shaped capability check only for AutoStrategy; an explicit ProviderStrategy bypasses auto-detection and lets native schema binding or request construction fail when it cannot be applied. AgentScope keeps the check coarse at the model level (supportsNativeStructuredOutput and its WithTools variant), with runtime fallback when native execution fails.

Could Flink expose only one request-shaped method on the base connection?

• AUTO calls supportsNativeStructuredOutput(schema, tools, modelParams) to choose native or prompt.
• NATIVE directly enters the native request path.
• PROMPT uses the existing prompt path.

The native request builder can still fail fast on deterministic incompatibilities—such as an unencodable schema, an unsupported API version, or conflicting request configuration—before sending anything to the provider. Effective-model resolution and the separate capability/feasibility checks could then remain provider-internal helpers, rather than three hooks on the common base class.

[wenjin272]

Non-blocking follow-up: RowTypeInfo does not necessarily have to remain on the prompt fallback permanently. Since OutputSchema already restricts it to basic field types, we could render a flat RowTypeInfo directly as JSON Schema in Java and Python, while leaving nested rows unsupported initially. This would allow it to use provider-native structured output without generating runtime POJO/Pydantic classes, and the result could still be parsed into a Flink Row.

[weiqingy]

Agreed that a flat RowTypeInfo is renderable without a runtime POJO or Pydantic class, since OutputSchema already restricts the fields to basic types, and the response still parses back into a Row. Nested rows are the part that would need a decision, so leaving those unsupported initially sounds right.

Worth doing as its own change rather than widening this one.

[purushah]

Thanks for the careful work here. I read both sides, ran the suites, and checked each override against its branch. It all lines up, and I couldn't find a valid request that behaves differently than before. I left a few comments on the code. None of them block this one.

What I verified

• Java on JDK 17: api 428, openai 131, anthropic 103, bedrock 90, gemini 71, ollama 18, watsonx 51. All green, matching the description.
• Python: 322 passed, 1 skipped across the seven changed test files. Pinned ruff is clean.
• Every override reads without consuming, is null-safe, and matches its branch's conditions. The only load-bearing raw-parameters pass is Java Anthropic, and it is wired correctly.

On the open thread

On the two-predicates question: I think the asymmetry argument is right, and vLLM only overriding capability is a real reason not to merge them. If a combining method comes later it has to take the strategy into account too, something like feasible && strategy.resolvesToNative(capable), otherwise NATIVE still can't override capability. That keeps the distinction you two already worked out in the thread. Happy either way, that's for @wenjin272 to decide.

---

Looks good to me overall. Everything below is non-blocking for this extraction.

[purushah]

Could we retain one TODO for #912 here noting that strategy resolution is not wired into production yet? When it is wired, the native branches need to honor the resolved policy without independently vetoing NATIVE through another capability check. Non-blocking for this extraction, it's unfinished #912 integration rather than something this PR changed.

Suggested change

	* #resolvesToNative(boolean)} combines the two.
	* #resolvesToNative(boolean)} combines the two.
	*
	* TODO(#912): strategy resolution is not wired into production yet. Once it is, the
	* native branches must honor the resolved policy rather than vetoing NATIVE through
	* their own capability check.

[purushah]

Ollama's capability check is always true, so this test can't tell if someone folds capability into the override later, both sides move together and it still passes. The Python test gets around that with an _IncapableConnection subclass that returns false for capability and checks the hook still says yes for a Pydantic model. Worth doing the same here with a POJO.

[purushah]

Same as Ollama, capability is always true here so there's no way to test that it isn't folded into the hook. These two are the ones the "dedicated test per connection" line in the description doesn't cover yet.

[purushah]

Small thing on the description. Tongyi and Azure use plain set membership for capability, so a non-string model like 123 was already returning False quietly before this change, only unhashable values raised. OpenAI did raise, at its containment check, and Anthropic at .startswith. Might be worth softening "four connections raised" and adding one test for the new fallback so it's pinned. Not a real risk in practice since the setup declares model: str.

[weiqingy]

Thanks for the review, and for running both suites. All six are addressed in dc8bfb03, along with a description fix.

One correction worth surfacing: on the non-string model it's three connections, not four. vLLM raises too. It inherits the query from OpenAI, so it didn't show up in the first sweep.

The one I left alone is the raw_kwargs copy. It keeps all six connections reading the same way, though it's a no-op on that path today. Does that trade seem worth it to you?

[purushah]

This is now the same function in anthropic, ollama and openai, and azure, tongyi and watsonx have their own versions too. If it lived next to render_output_schema in types.py, the schema-form check would have one home when the flat RowTypeInfo follow-up comes. Optional, fine as a follow-up.

[purushah]

Nothing gets popped before the hook on this path, so this copy is the same map as kwargs. The other connections do strip keys before an overridable hook, so their snapshots make sense as a contract for subclasses. Here it could just pass kwargs. Tiny, non-blocking.