diff --git a/python/README.md b/python/README.md index f3c62b91..02765944 100644 --- a/python/README.md +++ b/python/README.md @@ -68,17 +68,35 @@ Existing API-key and environment-based calls are unchanged. See the and [configured-client example](../site/src/docs/python/basic-usage.md#reusing-an-already-configured-client) for ownership, retry, and error behavior. +## Using the Responses API + +For new text and image integrations, pass `use_responses_api=True` to any model +factory or to `create_openai_language_model_from_client`. HTTP factories need +the complete Responses endpoint and a model/deployment name; Azure callers +supply that name with the `model` keyword. Configured clients need only +`responses.create` and remain application-owned, without a required SDK +dependency. + +Existing calls still use Chat Completions. Endpoint paths do not select an API, +and automatic Responses-to-Chat fallback is deferred. Both APIs preserve +TypeChat's schema validation and bounded repair. + +See the [Responses API guide](../site/src/docs/python/basic-usage.md#using-the-responses-api) +and [minimal HTTP/client example](./examples/responses/demo.py). + All four model factories accept a keyword-only `temperature` option. It defaults -to `0.0`, preserving existing behavior. Pass `temperature=None` to omit the -parameter entirely and use the configured client's or service's default, for -example with reasoning models that reject explicit sampling parameters. This -does not disable reasoning, change authentication, or select a different API. +to `0.0` for Chat Completions, preserving existing behavior. In Chat Completions +mode, pass `temperature=None` to omit the parameter entirely and use the +configured client's or service's default, for example with reasoning models +that reject explicit sampling parameters. Responses requests always omit +temperature, regardless of this option. This does not disable reasoning, change +authentication, or select a different API. ## Text and image prompts -Both built-in Chat Completions models and the configured-client adapter accept -typed text/image parts in `PromptSection.content`. Existing whole-string prompts -remain supported without migration. +Both built-in HTTP models and configured-client adapters accept typed text/image +parts in `PromptSection.content`, with Chat Completions or explicitly selected +Responses. Existing whole-string prompts remain supported without migration. The following example demonstrates both `model.complete` and translator context; applications can use either call independently. Supply your own hosted image URL @@ -127,6 +145,11 @@ bare strings. Text works in system, user, and assistant messages; images are supported only in user messages. Malformed input produces a content-free `Failure` locating the section or part before authentication or dispatch. +Use the same public prompt types for either API. Responses models map tagged +user/system text to `input_text`, assistant text to `output_text`, and images to +`input_image` with a flat `image_url` string and optional `detail`, preserving +order, references, and omitted detail. + `translate` still takes a string request. Its preamble stays before the generated schema/request message, and image context is preserved during validation and repair. Successful model completions remain text-only. Authentication, HTTP controls, and @@ -138,7 +161,6 @@ accessibility, image counts, detail support, and size/resolution limits depend o your provider and model. No image-processing or provider SDK dependency is added. See [text and image usage](../site/src/docs/python/basic-usage.md#text-and-image-prompts) for the complete prompt type definitions and limitations. - ## Documenting schema fields Trailing Python comments on directly declared `TypedDict` and dataclass fields become diff --git a/python/examples/README.md b/python/examples/README.md index 015064f0..f8becafb 100644 --- a/python/examples/README.md +++ b/python/examples/README.md @@ -11,6 +11,7 @@ We recommend reading each example in the following order. | Name | Description | | ---- | ----------- | | [Sentiment](https://github.com/microsoft/TypeChat/tree/main/python/examples/sentiment) | A sentiment classifier which categorizes user input as negative, neutral, or positive. This is TypeChat's "hello world!" | +| [Responses](./responses/demo.py) | Explicit Responses API selection for a one-shot HTTP request or an application-owned asynchronous client. | | [Coffee Shop](https://github.com/microsoft/TypeChat/tree/main/python/examples/coffeeShop) | An intelligent agent for a coffee shop. This sample translates user intent is translated to a list of coffee order items. | [Calendar](https://github.com/microsoft/TypeChat/tree/main/python/examples/calendar) | An intelligent scheduler. This sample translates user intent into a sequence of actions to modify a calendar. | | [HealthData](https://github.com/microsoft/TypeChat/tree/main/python/examples/healthData) | The Health Data Agent shows how strongly typed **agents with history** could interact with a user to collect information needed for one or more data types ("form filling"). | @@ -118,6 +119,22 @@ python examples/sentiment/non_interactive.py "TypeChat is awesome!" For an explanation of how to call the same async function from an existing backend, see the [Python quickstart](../../site/src/docs/python/non-interactive.md). +For a new text-only Responses integration, set `OPENAI_ENDPOINT` to the complete +Responses URL (for example `https://api.openai.com/v1/responses`) in addition to +`OPENAI_API_KEY` and `OPENAI_MODEL`, then run: + +```sh +python examples/responses/demo.py "TypeChat is awesome!" +``` + +For Azure, use `AZURE_OPENAI_API_KEY`, the complete `AZURE_OPENAI_ENDPOINT` supported +by your resource/API version, and `--model your-deployment`. The example also +exports `classify_with_client` for an application-owned asynchronous Responses +client, without requiring an SDK dependency. See the +[Responses API guide](../../site/src/docs/python/basic-usage.md#using-the-responses-api) +for supported text/output contracts, authentication, limits, and deferred fallback. +The existing examples retain their Chat Completions defaults. + To run an example interactively, type `python examples//demo.py` from the example's directory and enter requests when prompted. Type `quit` or `exit` to end the session. You can also open in VS Code the selected example's directory and press F5 to launch it in debug mode. Note that there are various sample "prose" files (e.g. `input.txt`) provided in each `src` directory that can give a sense of what you can run. diff --git a/python/examples/responses/demo.py b/python/examples/responses/demo.py new file mode 100644 index 00000000..f1d52be1 --- /dev/null +++ b/python/examples/responses/demo.py @@ -0,0 +1,49 @@ +import argparse +import asyncio +import json +import os +from dataclasses import asdict, dataclass +from typing import Literal + +from dotenv import dotenv_values +from typechat import ( + Failure, + ResponsesClient, + Result, + TypeChatJsonTranslator, + TypeChatLanguageModel, + TypeChatValidator, + create_language_model, + create_openai_language_model_from_client, +) + + +@dataclass +class Sentiment: + sentiment: Literal["negative", "neutral", "positive"] + + +async def classify(text: str, model: TypeChatLanguageModel) -> Result[Sentiment]: + translator = TypeChatJsonTranslator(model, TypeChatValidator(Sentiment), Sentiment) + return await translator.translate(text) + + +async def classify_with_client(text: str, client: ResponsesClient, model: str) -> Result[Sentiment]: + return await classify(text, create_openai_language_model_from_client(client, model, use_responses_api=True)) + + +async def main(text: str, model_name: str | None = None) -> None: + settings = {**dotenv_values(), **os.environ} + async with create_language_model(settings, use_responses_api=True, model=model_name) as model: + result = await classify(text, model) + if isinstance(result, Failure): + raise RuntimeError(result.message) + print(json.dumps(asdict(result.value))) + + +if __name__ == "__main__": + parser = argparse.ArgumentParser(description="Classify text using an explicitly selected Responses API.") + parser.add_argument("text", help="The text to classify.") + parser.add_argument("--model", help="Explicit model or Azure deployment name (required for Azure).") + args = parser.parse_args() + asyncio.run(main(args.text, args.model)) diff --git a/python/src/typechat/__init__.py b/python/src/typechat/__init__.py index 13ace520..13c9e6c5 100644 --- a/python/src/typechat/__init__.py +++ b/python/src/typechat/__init__.py @@ -4,6 +4,7 @@ from typechat._internal.model import ( ChatCompletionClient, + ResponsesClient, ImagePromptContent, ImageUrl, PromptContent, @@ -37,6 +38,7 @@ "ImageUrl", "TokenProvider", "ChatCompletionClient", + "ResponsesClient", "create_language_model", "create_openai_language_model", "create_azure_openai_language_model", diff --git a/python/src/typechat/_internal/model.py b/python/src/typechat/_internal/model.py index 44f87d15..df49fbf4 100644 --- a/python/src/typechat/_internal/model.py +++ b/python/src/typechat/_internal/model.py @@ -1,7 +1,7 @@ import asyncio import json import re -from collections.abc import Awaitable, Callable, Sequence +from collections.abc import Awaitable, Callable, Mapping, Sequence from copy import deepcopy from inspect import isawaitable from types import TracebackType @@ -87,6 +87,41 @@ def _normalize_prompt(prompt: object) -> Result[list[PromptSection]]: return Success(deepcopy(cast(list[PromptSection], prompt))) +class _ResponsesTextContent(TypedDict): + type: Literal["input_text", "output_text"] + text: str + +class _ResponsesImageContent(TypedDict): + type: Literal["input_image"] + image_url: str + detail: NotRequired[Literal["auto", "low", "high"]] + +class _ResponsesPromptSection(TypedDict): + role: Literal["system", "user", "assistant"] + content: str | list[_ResponsesTextContent | _ResponsesImageContent] + +def _responses_input(prompt: list[PromptSection]) -> list[_ResponsesPromptSection]: + messages: list[_ResponsesPromptSection] = [] + for section in prompt: + content = section["content"] + if isinstance(content, str): + messages.append({"role": section["role"], "content": content}) + continue + parts: list[_ResponsesTextContent | _ResponsesImageContent] = [] + for part in content: + if part["type"] == "text": + parts.append({ + "type": "output_text" if section["role"] == "assistant" else "input_text", + "text": part["text"], + }) + else: + image: _ResponsesImageContent = {"type": "input_image", "image_url": part["image_url"]["url"]} + if "detail" in part["image_url"]: + image["detail"] = part["image_url"]["detail"] + parts.append(image) + messages.append({"role": section["role"], "content": parts}) + return messages + TokenProvider = Callable[[], str | Awaitable[str]] class TypeChatLanguageModel(Protocol): @@ -128,6 +163,59 @@ class ChatCompletionClient(Protocol): @property def chat(self) -> _ClientChat: ... +class _ClientResponses(Protocol): + @property + def create(self) -> Callable[..., Awaitable[object]]: + # Validate canonical envelopes at runtime, whether mappings or SDK objects. + ... + +class ResponsesClient(Protocol): + """An asynchronous OpenAI-compatible Responses client supplied by the caller.""" + @property + def responses(self) -> _ClientResponses: ... + +def _response_field(value: object, name: str) -> object: + if isinstance(value, Mapping): + return cast(Mapping[str, object], value).get(name) + return getattr(value, name, None) + +def _responses_completion_result(response: object) -> Result[str]: + if _response_field(response, "status") != "completed": + return Failure("Responses API returned a response that was not completed.") + if _response_field(response, "error") is not None or _response_field(response, "incomplete_details") is not None: + return Failure("Responses API returned an error or incomplete response.") + output = _response_field(response, "output") + if not isinstance(output, list): + return Failure("Responses API returned a missing or malformed output list.") + + text_parts: list[str] = [] + for item in cast(list[object], output): + item_type = _response_field(item, "type") + if item_type == "reasoning": + if _response_field(item, "status") not in (None, "completed"): + return Failure("Responses API returned output that was not completed.") + continue + if item_type != "message" or _response_field(item, "role") != "assistant": + return Failure("Responses API returned unsupported output.") + if _response_field(item, "status") != "completed": + return Failure("Responses API returned output that was not completed.") + content = _response_field(item, "content") + if not isinstance(content, list) or not content: + return Failure("Responses API returned missing or malformed message content.") + for part in cast(list[object], content): + part_type = _response_field(part, "type") + if part_type in ("refusal", "blocked"): + return Failure("Responses API refused or blocked the request.") + if part_type != "output_text": + return Failure("Responses API returned unsupported message content.") + text = _response_field(part, "text") + if not isinstance(text, str): + return Failure("Responses API returned missing or malformed text content.") + text_parts.append(text) + if not text_parts: + return Failure("Responses API returned no text content.") + return Success("".join(text_parts)) + class _ClientLanguageModel(TypeChatLanguageModel): _client: ChatCompletionClient _model: str @@ -159,23 +247,51 @@ async def complete(self, prompt: str | list[PromptSection]) -> Result[str]: return Failure("Client completion returned no text content.") return Success(content) +class _ResponsesClientLanguageModel(TypeChatLanguageModel): + _client: ResponsesClient + _model: str + + def __init__(self, client: ResponsesClient, model: str): + super().__init__() + self._client = client + self._model = model + + @override + async def complete(self, prompt: str | list[PromptSection]) -> Result[str]: + normalized = _normalize_prompt(prompt) + if isinstance(normalized, Failure): + return normalized + response = await self._client.responses.create( + model=self._model, input=_responses_input(normalized.value), stream=False, + ) + return _responses_completion_result(response) + def create_openai_language_model_from_client( - client: ChatCompletionClient, + client: ChatCompletionClient | ResponsesClient, model: str, *, + use_responses_api: bool = False, temperature: float | None = 0.0, ) -> TypeChatLanguageModel: """ - Adapts an already configured asynchronous OpenAI-compatible Chat Completions client. + Adapts an already configured asynchronous OpenAI-compatible client. TypeChat does not import an SDK, reconfigure or close the client, or add retries or timeouts. Exceptions raised by the client propagate unchanged. The caller supplies the model name, - or the deployment name for Azure OpenAI. - Temperature defaults to 0.0; pass None to omit it and use the client's or service's default. + or the deployment name for Azure OpenAI. Set use_responses_api=True to call responses.create + instead of chat.completions.create. Only the selected API member is required. + Chat Completions temperature defaults to 0.0; pass None to omit it. + Responses requests omit temperature regardless of this option. """ if not model.strip(): raise ValueError("The client model or deployment name must not be empty.") - return _ClientLanguageModel(client, model, temperature) + if use_responses_api: + if not callable(_response_field(_response_field(client, "responses"), "create")): + raise ValueError("The configured client must provide responses.create.") + return _ResponsesClientLanguageModel(cast(ResponsesClient, client), model) + if not callable(_response_field(_response_field(_response_field(client, "chat"), "completions"), "create")): + raise ValueError("The configured client must provide chat.completions.create.") + return _ClientLanguageModel(cast(ChatCompletionClient, client), model, temperature) _TRANSIENT_ERROR_CODES = [ 429, @@ -212,6 +328,7 @@ class HttpxLanguageModel(TypeChatLanguageModel, AsyncContextManager): max_response_bytes: int = 100 * 1024 * 1024 _async_client: httpx.AsyncClient _token_provider: TokenProvider | None + _use_responses_api: bool def __init__( self, @@ -220,13 +337,25 @@ def __init__( default_params: dict[str, str], *, token_provider: TokenProvider | None = None, + use_responses_api: bool = False, temperature: float | None = 0.0, ): super().__init__() + if use_responses_api: + try: + endpoint = httpx.URL(url) + except httpx.InvalidURL: + raise ValueError("Responses API requires a complete HTTP or HTTPS endpoint.") from None + if not endpoint.is_absolute_url or endpoint.scheme not in ("http", "https"): + raise ValueError("Responses API requires a complete HTTP or HTTPS endpoint.") + model = default_params.get("model") + if model is None or not model.strip(): + raise ValueError("Responses API requires a nonempty model or deployment name.") self.url = url self.headers = headers self.default_params = default_params self._token_provider = token_provider + self._use_responses_api = use_responses_api self.temperature = temperature self._async_client = httpx.AsyncClient() @@ -240,14 +369,18 @@ async def complete(self, prompt: str | list[PromptSection]) -> Success[str] | Fa **self.headers, } - body = { - **self.default_params, - "messages": normalized.value, - "temperature": self.temperature, - "n": 1, - } - if self.temperature is None: - del body["temperature"] + body: dict[str, object] + if self._use_responses_api: + body = {"model": self.default_params["model"], "input": _responses_input(normalized.value), "stream": False} + else: + body = { + **self.default_params, + "messages": normalized.value, + "temperature": self.temperature, + "n": 1, + } + if self.temperature is None: + del body["temperature"] retry_count = 0 while True: try: @@ -268,6 +401,12 @@ async def complete(self, prompt: str | list[PromptSection]) -> Success[str] | Fa ) as response: if response.is_success: raw = await self._read_capped(response) + if self._use_responses_api: + try: + response_body: object = json.loads(raw) + except ValueError: + return Failure("Responses API returned malformed JSON.") + return _responses_completion_result(response_body) json_result = cast( dict[Literal["choices"], list[dict[Literal["message"], _HttpResponseMessage]]], json.loads(raw) @@ -280,11 +419,15 @@ async def complete(self, prompt: str | list[PromptSection]) -> Success[str] | Fa return Success(content) if response.status_code not in _TRANSIENT_ERROR_CODES or retry_count >= self.max_retry_attempts: + if self._use_responses_api: + return Failure(f"Responses API request failed with HTTP status {response.status_code}.") return Failure(f"REST API error {response.status_code}: {response.reason_phrase}") except _ResponseTooLargeError as e: return Failure(str(e)) except Exception as e: if retry_count >= self.max_retry_attempts: + if self._use_responses_api: + return Failure("Responses API request failed.") return Failure(str(e) or f"{repr(e)} raised from within internal TypeChat language model.") await asyncio.sleep(self.retry_pause_seconds) @@ -334,6 +477,8 @@ def __del__(self): def create_language_model( vals: dict[str, str | None], *, + use_responses_api: bool = False, + model: str | None = None, temperature: float | None = 0.0, ) -> HttpxLanguageModel: """ @@ -341,17 +486,22 @@ def create_language_model( chosen by a dictionary of variables (typically just `os.environ`). If an `OPENAI_API_KEY` environment variable exists, an OpenAI model is constructed. - The `OPENAI_ENDPOINT` and `OPENAI_MODEL` environment variables must also be defined or an error will be raised. + OPENAI_MODEL is required unless model is supplied explicitly. OPENAI_ENDPOINT defaults to + the Chat Completions endpoint, but must be supplied in Responses mode. If an `AZURE_OPENAI_API_KEY` environment variable exists, an Azure OpenAI model is constructed. - The `AZURE_OPENAI_ENDPOINT` environment variable must also be defined or an exception will be thrown. + AZURE_OPENAI_ENDPOINT is required. Responses mode also requires an explicit model/deployment name. If none of these key variables are defined, an exception is thrown. @returns An instance of `TypeChatLanguageModel`. Args: vals: A dictionary of variables. Typically just `os.environ`. - temperature: Sampling temperature, defaulting to 0.0. None omits the request parameter. + use_responses_api: Use Responses instead of Chat Completions. Requires a complete endpoint. + model: An optional explicit model or Azure deployment name. Required for Azure Responses; + otherwise OpenAI uses OPENAI_MODEL and legacy Azure requests omit the model. + temperature: Chat Completions sampling temperature, defaulting to 0.0. None omits it. + Responses requests always omit this parameter. """ def required_var(name: str) -> str: @@ -362,24 +512,29 @@ def required_var(name: str) -> str: if "OPENAI_API_KEY" in vals: api_key = required_var("OPENAI_API_KEY") - model = required_var("OPENAI_MODEL") - endpoint = vals.get("OPENAI_ENDPOINT", None) or "https://api.openai.com/v1/chat/completions" + model_name = model if model is not None else required_var("OPENAI_MODEL") + endpoint = vals.get("OPENAI_ENDPOINT", None) or None org = vals.get("OPENAI_ORGANIZATION", None) or vals.get("OPENAI_ORG", None) or "" - return create_openai_language_model(api_key, model, endpoint, org, temperature=temperature) + return create_openai_language_model( + api_key, model_name, endpoint, org, use_responses_api=use_responses_api, temperature=temperature, + ) elif "AZURE_OPENAI_API_KEY" in vals: api_key=required_var("AZURE_OPENAI_API_KEY") endpoint=required_var("AZURE_OPENAI_ENDPOINT") - return create_azure_openai_language_model(api_key, endpoint, temperature=temperature) + return create_azure_openai_language_model( + api_key, endpoint, model=model, use_responses_api=use_responses_api, temperature=temperature, + ) else: raise ValueError("Missing environment variables for OPENAI_API_KEY or AZURE_OPENAI_API_KEY.") def create_openai_language_model( api_key: str, model: str, - endpoint: str = "https://api.openai.com/v1/chat/completions", + endpoint: str | None = None, org: str = "", *, + use_responses_api: bool = False, temperature: float | None = 0.0, ) -> HttpxLanguageModel: """ @@ -388,10 +543,17 @@ def create_openai_language_model( Args: api_key: The OpenAI API key. model: The OpenAI model name. - endpoint: The OpenAI REST API endpoint. + endpoint: The complete OpenAI REST API endpoint. Required in Responses mode; + otherwise defaults to https://api.openai.com/v1/chat/completions. org: The OpenAI organization. - temperature: Sampling temperature, defaulting to 0.0. None omits the request parameter. + use_responses_api: Use Responses instead of Chat Completions, without endpoint inference or fallback. + temperature: Chat Completions sampling temperature, defaulting to 0.0. None omits it. + Responses requests always omit this parameter. """ + if endpoint is None: + if use_responses_api: + raise ValueError("Responses API requires an explicit endpoint.") + endpoint = "https://api.openai.com/v1/chat/completions" headers = { "Authorization": f"Bearer {api_key}", "OpenAI-Organization": org, @@ -400,7 +562,8 @@ def create_openai_language_model( "model": model, } return HttpxLanguageModel( - url=endpoint, headers=headers, default_params=default_params, temperature=temperature, + url=endpoint, headers=headers, default_params=default_params, use_responses_api=use_responses_api, + temperature=temperature, ) def create_azure_openai_language_model( @@ -408,6 +571,8 @@ def create_azure_openai_language_model( endpoint: str, *, token_provider: TokenProvider | None = None, + use_responses_api: bool = False, + model: str | None = None, temperature: float | None = 0.0, ) -> HttpxLanguageModel: """ @@ -418,13 +583,18 @@ def create_azure_openai_language_model( endpoint: The Azure OpenAI REST API endpoint. token_provider: An optional synchronous or asynchronous bearer-token provider. Invoked for each request attempt; the provider is responsible for caching and renewal. - temperature: Sampling temperature, defaulting to 0.0. None omits the request parameter. + use_responses_api: Use Responses instead of Chat Completions, without endpoint inference or fallback. + model: The model or deployment name. Required in Responses mode; omitted from legacy Azure bodies. + temperature: Chat Completions sampling temperature, defaulting to 0.0. None omits it. + Responses requests always omit this parameter. """ + default_params = {"model": model} if model is not None else {} if token_provider is not None: if api_key is not None: raise ValueError("Specify either an API key or a token provider, not both.") return HttpxLanguageModel( - url=endpoint, headers={}, default_params={}, token_provider=token_provider, temperature=temperature, + url=endpoint, headers={}, default_params=default_params, + token_provider=token_provider, use_responses_api=use_responses_api, temperature=temperature, ) if api_key is None: raise ValueError("Specify an API key or a token provider.") @@ -434,4 +604,7 @@ def create_azure_openai_language_model( # Needed when using regular API key "api-key": api_key, } - return HttpxLanguageModel(url=endpoint, headers=headers, default_params={}, temperature=temperature) + return HttpxLanguageModel( + url=endpoint, headers=headers, default_params=default_params, use_responses_api=use_responses_api, + temperature=temperature, + ) diff --git a/python/tests/test_multimodal_model.py b/python/tests/test_multimodal_model.py index 340f2ba3..e2fe7ab1 100644 --- a/python/tests/test_multimodal_model.py +++ b/python/tests/test_multimodal_model.py @@ -240,7 +240,7 @@ def _assert_invalid_prompt( harness.client.chat.completions.create.assert_not_called() -_MALFORMED_PROMPTS: list[tuple[object, str]] = [ +MALFORMED_PROMPTS: list[tuple[object, str]] = [ (None, "Prompt"), (42, "Prompt"), (b"text", "Prompt"), @@ -261,7 +261,7 @@ def _assert_invalid_prompt( ] -@pytest.mark.parametrize("prompt, location", _MALFORMED_PROMPTS) +@pytest.mark.parametrize("prompt, location", MALFORMED_PROMPTS) def test_rejects_malformed_sections_before_authentication_or_dispatch( model_factory: ModelFactory, prompt: object, @@ -272,7 +272,7 @@ def test_rejects_malformed_sections_before_authentication_or_dispatch( _assert_invalid_prompt(harness, result, location) -@pytest.mark.parametrize("part", [ +MALFORMED_PARTS: list[object] = [ _PRIVATE_TEXT, None, 42, @@ -298,7 +298,10 @@ def test_rejects_malformed_sections_before_authentication_or_dispatch( {"type": "image_url", "image_url": {"url": _DATA_IMAGE, "detail": None}}, {"type": "image_url", "image_url": {"url": _HOSTED_IMAGE, "detail": []}}, {"type": "image_url", "image_url": {"url": _HOSTED_IMAGE, "detail": 42}}, -]) +] + + +@pytest.mark.parametrize("part", MALFORMED_PARTS) def test_rejects_malformed_parts_without_exposing_content(model_factory: ModelFactory, part: object): prompt = [ {"role": "system", "content": "Instructions"}, diff --git a/python/tests/test_prompt_types.py b/python/tests/test_prompt_types.py index 5a427a2f..9e472ca3 100644 --- a/python/tests/test_prompt_types.py +++ b/python/tests/test_prompt_types.py @@ -10,6 +10,7 @@ ImageUrl, PromptContent, PromptSection, + ResponsesClient, Result, TextPromptContent, TypeChatJsonTranslator, @@ -84,10 +85,15 @@ async def check_completion_and_request_types( model: TypeChatLanguageModel, translator: TypeChatJsonTranslator[ImageDescription], client: ChatCompletionClient, + responses_client: ResponsesClient, ) -> None: sections: list[PromptSection] = [{"role": "user", "content": "existing text"}] assert_type(await model.complete("existing text"), Result[str]) assert_type(await model.complete(sections), Result[str]) + responses_model = create_openai_language_model_from_client( + responses_client, "model", use_responses_api=True, + ) + assert_type(await inspect_images(responses_model, "aW1hZ2U="), tuple[Result[str], Result[ImageDescription]]) assert_type(await translator.translate("existing text"), Result[ImageDescription]) assert_type( await translator.translate("existing text", prompt_preamble="context"), diff --git a/python/tests/test_responses_model.py b/python/tests/test_responses_model.py new file mode 100644 index 00000000..382bf477 --- /dev/null +++ b/python/tests/test_responses_model.py @@ -0,0 +1,933 @@ +import asyncio +import json +import subprocess +import sys +from collections.abc import AsyncIterator, Awaitable, Callable, Iterator, Sequence +from contextlib import asynccontextmanager +from copy import deepcopy +from dataclasses import dataclass +from inspect import Parameter, signature +from types import SimpleNamespace +from unittest.mock import AsyncMock, Mock +from typing_extensions import Literal, NotRequired, TypedDict, cast + +import httpx +import pytest +import typechat +from typechat._internal.model import HttpxLanguageModel + +from .test_multimodal_model import MALFORMED_PARTS, MALFORMED_PROMPTS + + +Handler = Callable[[httpx.Request], httpx.Response] +MockClient = Callable[[Handler], httpx.AsyncClient] + + +@pytest.fixture +def mock_client(monkeypatch: pytest.MonkeyPatch) -> MockClient: + async_client = httpx.AsyncClient + + def create(handler: Handler) -> httpx.AsyncClient: + client = async_client(transport=httpx.MockTransport(handler)) + monkeypatch.setattr(httpx, "AsyncClient", lambda: client) + return client + + return create + + +class _InputText(TypedDict): + type: Literal["input_text", "output_text"] + text: str + + +class _InputImage(TypedDict): + type: Literal["input_image"] + image_url: str + detail: NotRequired[Literal["auto", "low", "high"]] + + +class _InputSection(TypedDict): + role: Literal["system", "user", "assistant"] + content: str | list[_InputText | _InputImage] + + +class _Request(TypedDict): + model: str + input: list[_InputSection] + stream: Literal[False] + + +class _Responses: + requests: list[_Request] + _outputs: Iterator[object] + + def __init__(self, outputs: Sequence[object]): + super().__init__() + self.requests = [] + self._outputs = iter(outputs) + + async def create( + self, *, model: str, input: list[_InputSection], stream: Literal[False], + ) -> object: + self.requests.append({"model": model, "input": deepcopy(input), "stream": stream}) + output = next(self._outputs) + if isinstance(output, BaseException): + raise output + return output + + +@dataclass +class _Client: + responses: _Responses + timeout: float = 42.0 + closed: bool = False + + async def close(self) -> None: + self.closed = True + + +def _message(content: object, *, status: object = "completed", role: object = "assistant") -> dict[str, object]: + return {"type": "message", "role": role, "status": status, "content": content} + + +def _text(text: object) -> dict[str, object]: + return {"type": "output_text", "text": text, "annotations": []} + + +def _response(text: str = "Hello!") -> dict[str, object]: + return { + "status": "completed", + "output": [_message([_text(text)])], + "error": None, + "incomplete_details": None, + } + + +def _sdk_object(value: object) -> object: + if isinstance(value, dict): + return SimpleNamespace(**{key: _sdk_object(item) for key, item in cast(dict[str, object], value).items()}) + if isinstance(value, list): + return [_sdk_object(item) for item in cast(list[object], value)] + return value + + +def _model( + outputs: Sequence[object], use_client: bool, mock_client: MockClient, + *, temperature: float | None = 0.0, token_provider: typechat.TokenProvider | None = None, +) -> tuple[typechat.TypeChatLanguageModel, list[_Request]]: + if use_client: + client = _Client(_Responses([_sdk_object(output) for output in outputs])) + configured: typechat.ResponsesClient = client + return ( + typechat.create_openai_language_model_from_client( + configured, "my-deployment", use_responses_api=True, temperature=temperature, + ), + client.responses.requests, + ) + remaining = iter(outputs) + requests: list[_Request] = [] + + def handler(request: httpx.Request) -> httpx.Response: + requests.append(cast(_Request, json.loads(request.content))) + return httpx.Response(200, json=next(remaining)) + + mock_client(handler) + endpoint = "https://example.invalid/opaque?api-version=test" + model = ( + typechat.create_azure_openai_language_model( + None, endpoint, model="my-deployment", use_responses_api=True, + token_provider=token_provider, temperature=temperature, + ) + if token_provider is not None else + typechat.create_openai_language_model( + "test-key", "my-deployment", endpoint, use_responses_api=True, temperature=temperature, + ) + ) + model.retry_pause_seconds = 0 + return model, requests + + +@asynccontextmanager +async def _open_model(model: typechat.TypeChatLanguageModel) -> AsyncIterator[typechat.TypeChatLanguageModel]: + if isinstance(model, HttpxLanguageModel): + async with model: + yield model + else: + yield model + + +def _complete( + model: typechat.TypeChatLanguageModel, prompt: str | list[typechat.PromptSection] = "hello", +) -> typechat.Result[str]: + async def run() -> typechat.Result[str]: + async with _open_model(model): + return await model.complete(prompt) + return asyncio.run(run()) + + +def _image_prompts( + url: str = "https://images.example.invalid/photo.png?ref=exact%2Fvalue", + detail: Literal["auto", "low", "high"] | None = None, +) -> tuple[list[typechat.PromptSection], list[_InputSection]]: + image: typechat.ImageUrl = {"url": url} + expected_image: _InputImage = {"type": "input_image", "image_url": url} + if detail is not None: + image["detail"] = detail + expected_image["detail"] = detail + data_url = "data:image/png;base64,aW1hZ2U=" + return ( + [ + {"role": "system", "content": [{"type": "text", "text": "Follow the schema."}]}, + {"role": "user", "content": [ + {"type": "text", "text": "First image:"}, + {"type": "image_url", "image_url": image}, + {"type": "text", "text": "Second image:"}, + {"type": "image_url", "image_url": {"url": data_url, "detail": "high"}}, + ]}, + {"role": "assistant", "content": [{"type": "text", "text": "Ready."}]}, + ], + [ + {"role": "system", "content": [{"type": "input_text", "text": "Follow the schema."}]}, + {"role": "user", "content": [ + {"type": "input_text", "text": "First image:"}, + expected_image, + {"type": "input_text", "text": "Second image:"}, + {"type": "input_image", "image_url": data_url, "detail": "high"}, + ]}, + {"role": "assistant", "content": [{"type": "output_text", "text": "Ready."}]}, + ], + ) + + +@pytest.mark.parametrize("use_client", [False, True]) +@pytest.mark.parametrize("temperature", [0.0, 1.0, None]) +def test_responses_preserve_text_roles_and_send_only_responses_fields( + use_client: bool, temperature: float | None, mock_client: MockClient, +): + model, requests = _model([_response()], use_client, mock_client, temperature=temperature) + prompt: list[typechat.PromptSection] = [ + {"role": "system", "content": "instructions"}, + {"role": "user", "content": "question"}, + {"role": "assistant", "content": "previous answer"}, + ] + original = [message.copy() for message in prompt] + assert _complete(model, prompt) == typechat.Success("Hello!") + assert requests == [{"model": "my-deployment", "input": original, "stream": False}] + assert prompt == original + + +@pytest.mark.parametrize("use_client", [False, True]) +@pytest.mark.parametrize("detail", [None, "auto", "low", "high"]) +@pytest.mark.parametrize("url", [ + "https://images.example.invalid/photo.png?ref=exact%2Fvalue", + "data:image/png;base64,aW1hZ2U=", +]) +def test_responses_maps_mixed_parts_and_preserves_images_and_optional_detail( + use_client: bool, detail: Literal["auto", "low", "high"] | None, url: str, mock_client: MockClient, +): + model, requests = _model([_response()], use_client, mock_client) + prompt, expected = _image_prompts(url, detail) + original = deepcopy(prompt) + assert _complete(model, prompt) == typechat.Success("Hello!") + assert requests == [{"model": "my-deployment", "input": expected, "stream": False}] + assert prompt == original + + +@pytest.mark.parametrize("use_client", [False, True]) +@pytest.mark.parametrize("role, tag", [ + ("system", "input_text"), ("user", "input_text"), ("assistant", "output_text"), +]) +def test_responses_tagged_text_uses_role_specific_parts( + use_client: bool, role: Literal["system", "user", "assistant"], + tag: Literal["input_text", "output_text"], mock_client: MockClient, +): + model, requests = _model([_response()], use_client, mock_client) + prompt: list[typechat.PromptSection] = [ + {"role": role, "content": [{"type": "text", "text": ""}, {"type": "text", "text": " Exact text "}]}, + ] + original = deepcopy(prompt) + assert _complete(model, prompt) == typechat.Success("Hello!") + assert requests[0]["input"] == [ + {"role": role, "content": [{"type": tag, "text": ""}, {"type": tag, "text": " Exact text "}]}, + ] + assert prompt == original + + +def test_responses_client_cannot_mutate_caller_owned_image_prompts(monkeypatch: pytest.MonkeyPatch): + client = _Client(_Responses([])) + prompt, expected = _image_prompts() + original = deepcopy(prompt) + + async def create(*, model: str, input: list[_InputSection], stream: Literal[False]) -> object: + assert model == "my-deployment" and stream is False + assert input == expected + content = input[1]["content"] + assert isinstance(content, list) + part = content[1] + assert part["type"] == "input_image" + part["image_url"] = "changed by the configured client" + content.clear() + input.clear() + return _response() + + monkeypatch.setattr(client.responses, "create", create) + model = typechat.create_openai_language_model_from_client(client, "my-deployment", use_responses_api=True) + assert _complete(model, prompt) == typechat.Success("Hello!") + assert prompt == original + assert not client.closed + + +@pytest.mark.parametrize("use_client", [False, True]) +@pytest.mark.parametrize("prompt, location", [ + *MALFORMED_PROMPTS, + *[ + ([{"role": "user", "content": [part]}], "section 0, part 0") + for part in MALFORMED_PARTS + ], + *[ + ([{"role": role, "content": [{"type": "image_url", "image_url": {"url": "private-image"}}]}], + "section 0, part 0") + for role in ("system", "assistant") + ], +]) +def test_responses_rejects_invalid_prompts_before_authentication_or_dispatch( + use_client: bool, prompt: object, location: str, mock_client: MockClient, +): + provider = Mock(return_value="test-token") + model, requests = _model([], use_client, mock_client, token_provider=provider) + result = _complete(model, cast(str | list[typechat.PromptSection], prompt)) + assert isinstance(result, typechat.Failure) + assert location in result.message + assert "private" not in result.message + assert "prompt-data-must-not-leak" not in result.message + assert "data:image" not in result.message + assert "https://" not in result.message + assert not result.has_value and result.response_text is None + assert requests == [] + provider.assert_not_called() + + +@pytest.mark.parametrize("use_client", [False, True]) +def test_normalizes_string_input_without_changing_the_selected_api(use_client: bool, mock_client: MockClient): + model, requests = _model([_response()], use_client, mock_client) + assert _complete(model) == typechat.Success("Hello!") + assert requests[0]["input"] == [{"role": "user", "content": "hello"}] + + +@pytest.mark.parametrize("use_client", [False, True]) +def test_concatenates_all_text_in_output_order_and_ignores_reasoning(use_client: bool, mock_client: MockClient): + payload: dict[str, object] = { + **_response(), + "output": [ + {"type": "reasoning", "summary": [], "encrypted_content": "not text"}, + _message([_text("one"), _text("two")]), + {"type": "reasoning", "status": "completed", "summary": []}, + _message([_text("three")]), + ], + "output_text": "must not override canonical output", + "unrelated": {"future": "field"}, + } + model, _ = _model([payload], use_client, mock_client) + assert _complete(model) == typechat.Success("onetwothree") + + +@pytest.mark.parametrize("use_client", [False, True]) +def test_present_empty_text_is_not_missing_text(use_client: bool, mock_client: MockClient): + model, _ = _model([_response("")], use_client, mock_client) + assert _complete(model) == typechat.Success("") + + +_INVALID_OUTPUTS: list[object] = [ + None, + [], + {}, + {"output_text": "not a canonical envelope"}, + {"status": "completed", "output_text": "not canonical"}, + *[{**_response(), "status": status} for status in [ + None, "failed", "incomplete", "cancelled", "in_progress", "queued", "unknown", + ]], + {**_response(), "error": {"message": "provider-payload-marker"}}, + {**_response(), "error": {}}, + {**_response(), "incomplete_details": {"reason": "content_filter"}}, + {**_response(), "output": None}, + {**_response(), "output": {}}, + {**_response(), "output": "not a list"}, + {**_response(), "output": []}, + {**_response(), "output": [{"type": "reasoning", "summary": []}]}, + {**_response(), "output": [None]}, + {**_response(), "output": [_message(None)]}, + {**_response(), "output": [_message([])]}, + {**_response(), "output": [_message({})]}, + {**_response(), "output": [_message([_text("partial")], status=None)]}, + {**_response(), "output": [_message([_text("partial")], status="incomplete")]}, + {**_response(), "output": [_message([_text("partial")], status="in_progress")]}, + {**_response(), "output": [_message([_text("partial")], role="user")]}, + {**_response(), "output": [_message([None])]}, + {**_response(), "output": [_message([dict[str, object]()])]}, + {**_response(), "output": [_message([{"type": "output_text"}])]}, + {**_response(), "output": [_message([_text(None)])]}, + {**_response(), "output": [_message([_text(42)])]}, + {**_response(), "output": [_message([_text("partial"), {"type": "refusal", "refusal": "provider-payload-marker"}])]}, + {**_response(), "output": [_message([_text("partial"), {"type": "blocked"}])]}, + {**_response(), "output": [_message([_text("partial"), {"type": "output_image"}])]}, + {**_response(), "output": [_message([_text("partial")]), {"type": "function_call"}]}, + {**_response(), "output": [_message([_text("partial")]), {"type": "web_search_call"}]}, + {**_response(), "output": [_message([_text("partial")]), {"type": "reasoning", "status": "incomplete"}]}, +] + + +@pytest.mark.parametrize("use_client", [False, True]) +@pytest.mark.parametrize("payload", _INVALID_OUTPUTS) +def test_malformed_refused_or_unsupported_output_fails_without_retry_or_fallback( + payload: object, use_client: bool, mock_client: MockClient, +): + model, requests = _model([payload], use_client, mock_client) + result = _complete(model) + assert isinstance(result, typechat.Failure) + assert "Responses API" in result.message + assert "provider-payload-marker" not in result.message + assert not result.has_value + assert result.response_text is None + assert len(requests) == 1 + + +def test_sdk_convenience_property_is_never_used(): + class Response: + status = "completed" + output: list[object] = [] + + @property + def output_text(self) -> str: + pytest.fail("The convenience property must not bypass canonical validation.") + + client = _Client(_Responses([Response()])) + model = typechat.create_openai_language_model_from_client(client, "model", use_responses_api=True) + assert isinstance(_complete(model), typechat.Failure) + + +def test_responses_only_client_is_reused_and_remains_caller_owned(): + client = _Client(_Responses([_sdk_object(_response("first")), _sdk_object(_response("second"))])) + model = typechat.create_openai_language_model_from_client(client, "my-deployment", use_responses_api=True) + + async def run() -> None: + assert await model.complete("one") == typechat.Success("first") + assert await model.complete("two") == typechat.Success("second") + + asyncio.run(run()) + assert len(client.responses.requests) == 2 + assert client.timeout == 42.0 + assert not client.closed + + +def test_configured_client_mappings_use_the_same_canonical_parser(): + client = _Client(_Responses([_response()])) + model = typechat.create_openai_language_model_from_client(client, "model", use_responses_api=True) + assert _complete(model) == typechat.Success("Hello!") + + +@pytest.mark.parametrize("error", [RuntimeError("client-marker"), asyncio.CancelledError()]) +def test_client_exceptions_and_cancellation_propagate_without_added_retries(error: BaseException): + client = _Client(_Responses([error])) + model = typechat.create_openai_language_model_from_client(client, "model", use_responses_api=True) + with pytest.raises(type(error)) as raised: + _complete(model) + assert raised.value is error + assert len(client.responses.requests) == 1 + assert not client.closed + + +def test_client_requires_only_the_selected_api_member(): + client = _Client(_Responses([])) + with pytest.raises(ValueError, match="chat.completions.create"): + typechat.create_openai_language_model_from_client(client, "model") + missing = cast(typechat.ResponsesClient, object()) + with pytest.raises(ValueError, match="responses.create"): + typechat.create_openai_language_model_from_client(missing, "model", use_responses_api=True) + + +@pytest.mark.parametrize("model_name", ["", " "]) +def test_client_requires_a_nonempty_model(model_name: str): + with pytest.raises(ValueError, match="must not be empty"): + typechat.create_openai_language_model_from_client( + _Client(_Responses([])), model_name, use_responses_api=True, + ) + + +@pytest.mark.parametrize("endpoint", [ + "https://example.invalid/v1/responses", + "https://example.invalid/openai/responses?api-version=test", + "https://example.invalid/openai/deployments/deployment/responses?api-version=test", + "https://example.invalid/opaque/chat/completions?route=responses&version=test", +]) +@pytest.mark.parametrize("azure", [False, True]) +@pytest.mark.parametrize("temperature", [0.0, 1.0, None]) +def test_http_factories_preserve_complete_destinations_and_model_placement( + endpoint: str, azure: bool, temperature: float | None, mock_client: MockClient, +): + requests: list[httpx.Request] = [] + + def handler(request: httpx.Request) -> httpx.Response: + requests.append(request) + return httpx.Response(200, json=_response()) + + mock_client(handler) + model = ( + typechat.create_azure_openai_language_model( + "literal-key", endpoint, model="deployment", use_responses_api=True, temperature=temperature, + ) + if azure else + typechat.create_openai_language_model( + "literal-key", "deployment", endpoint, "org", use_responses_api=True, temperature=temperature, + ) + ) + assert _complete(model) == typechat.Success("Hello!") + assert str(requests[0].url) == endpoint + assert json.loads(requests[0].content) == { + "model": "deployment", "input": [{"role": "user", "content": "hello"}], "stream": False, + } + if azure: + assert requests[0].headers["api-key"] == "literal-key" + else: + assert requests[0].headers["OpenAI-Organization"] == "org" + + +@pytest.mark.parametrize("endpoint", ["", "/responses", "not-an-address", "ftp://example.invalid/responses", "https://"]) +def test_responses_requires_an_absolute_http_destination(endpoint: str): + with pytest.raises(ValueError, match="complete HTTP or HTTPS endpoint"): + typechat.create_openai_language_model("key", "model", endpoint, use_responses_api=True) + + +def test_responses_does_not_silently_use_the_chat_default_endpoint(): + with pytest.raises(ValueError, match="explicit endpoint"): + typechat.create_openai_language_model("key", "model", use_responses_api=True) + with pytest.raises(ValueError, match="explicit endpoint"): + typechat.create_language_model({"OPENAI_API_KEY": "key", "OPENAI_MODEL": "model"}, use_responses_api=True) + + +def test_responses_selection_is_keyword_only_and_off_by_default_on_every_factory(): + for factory in [ + typechat.create_language_model, typechat.create_openai_language_model, + typechat.create_azure_openai_language_model, typechat.create_openai_language_model_from_client, + ]: + parameter = signature(factory).parameters["use_responses_api"] + assert parameter.kind == Parameter.KEYWORD_ONLY + assert parameter.default is False + + +@pytest.mark.parametrize("model_name", [None, "", " "]) +def test_azure_responses_requires_an_explicit_model_or_deployment(model_name: str | None): + with pytest.raises(ValueError, match="nonempty model or deployment"): + typechat.create_azure_openai_language_model( + "key", "https://example.invalid/opaque", model=model_name, use_responses_api=True, + ) + + +@pytest.mark.parametrize("model_name", ["", " "]) +def test_openai_responses_requires_a_nonempty_model(model_name: str): + with pytest.raises(ValueError, match="nonempty model or deployment"): + typechat.create_openai_language_model( + "key", model_name, "https://example.invalid/responses", use_responses_api=True, + ) + + +@pytest.mark.parametrize("azure", [False, True]) +@pytest.mark.parametrize("temperature", [0.0, 1.0, None]) +def test_dictionary_factory_supports_responses_without_new_environment_names( + azure: bool, temperature: float | None, mock_client: MockClient, +): + requests: list[httpx.Request] = [] + + def handler(request: httpx.Request) -> httpx.Response: + requests.append(request) + return httpx.Response(200, json=_response()) + + mock_client(handler) + settings: dict[str, str | None] = ( + {"AZURE_OPENAI_API_KEY": "identity", "AZURE_OPENAI_ENDPOINT": "https://example.invalid/opaque?api-version=test"} + if azure else + { + "OPENAI_API_KEY": "key", "OPENAI_ENDPOINT": "https://example.invalid/opaque", "OPENAI_MODEL": "model", + "OPENAI_ORGANIZATION": "preferred-org", "OPENAI_ORG": "legacy-org", "AZURE_OPENAI_API_KEY": "ignored", + } + ) + model = typechat.create_language_model( + settings, model="deployment" if azure else None, use_responses_api=True, temperature=temperature, + ) + assert _complete(model) == typechat.Success("Hello!") + assert json.loads(requests[0].content)["model"] == ("deployment" if azure else "model") + assert "temperature" not in json.loads(requests[0].content) + if azure: + assert requests[0].headers["api-key"] == "identity" + else: + assert requests[0].headers["OpenAI-Organization"] == "preferred-org" + assert "api-key" not in requests[0].headers + + +def test_dictionary_factory_accepts_an_explicit_openai_model_override(mock_client: MockClient): + mock_client(lambda request: httpx.Response(200, json=_response())) + model = typechat.create_language_model( + {"OPENAI_API_KEY": "key", "OPENAI_ENDPOINT": "https://example.invalid/opaque", "OPENAI_MODEL": "unused"}, + model="selected", use_responses_api=True, + ) + assert model.default_params == {"model": "selected"} + assert _complete(model) == typechat.Success("Hello!") + + +def test_endpoint_spelling_does_not_change_legacy_chat_mode(mock_client: MockClient): + requests: list[httpx.Request] = [] + + def handler(request: httpx.Request) -> httpx.Response: + requests.append(request) + return httpx.Response(200, json={"choices": [{"message": {"content": "legacy"}}]}) + + mock_client(handler) + model = typechat.create_openai_language_model("key", "model", "https://example.invalid/responses") + assert _complete(model) == typechat.Success("legacy") + assert json.loads(requests[0].content) == { + "model": "model", "messages": [{"role": "user", "content": "hello"}], "temperature": 0.0, "n": 1, + } + + +@pytest.mark.parametrize("provider_kind", ["sync", "async", "awaitable"]) +@pytest.mark.parametrize("with_images", [False, True]) +def test_tokens_resolve_afresh_for_responses_retries_and_completion( + provider_kind: str, with_images: bool, mock_client: MockClient, +): + requests: list[httpx.Request] = [] + statuses = iter([429, 500, 503, 200, 200]) + token_count = 0 + + def next_token() -> str: + nonlocal token_count + token_count += 1 + return f"token-{token_count}" + + async def async_provider() -> str: + await asyncio.sleep(0) + return next_token() + + def awaitable_provider() -> Awaitable[str]: + future: asyncio.Future[str] = asyncio.get_running_loop().create_future() + future.set_result(next_token()) + return future + + def handler(request: httpx.Request) -> httpx.Response: + requests.append(request) + return httpx.Response(next(statuses), json=_response()) + + mock_client(handler) + providers: dict[str, typechat.TokenProvider] = { + "sync": next_token, "async": async_provider, "awaitable": awaitable_provider, + } + model = typechat.create_azure_openai_language_model( + None, "https://example.invalid/opaque", model="deployment", use_responses_api=True, + token_provider=providers[provider_kind], + ) + assert (model.max_retry_attempts, model.retry_pause_seconds, model.timeout_seconds, model.max_response_bytes) == ( + 3, 1.0, 10, 100 * 1024 * 1024, + ) + model.retry_pause_seconds = 0 + model.timeout_seconds = 7 + image_prompt, expected_images = _image_prompts() + + async def run() -> None: + async with model: + assert await model.complete(image_prompt if with_images else "first") == typechat.Success("Hello!") + assert await model.complete(image_prompt if with_images else "second") == typechat.Success("Hello!") + + asyncio.run(run()) + assert token_count == len(requests) == 5 + assert [request.headers["Authorization"].removeprefix("Bearer ") for request in requests] == [ + f"token-{number}" for number in range(1, 6) + ] + assert all("api-key" not in request.headers for request in requests) + if with_images: + assert all(json.loads(request.content)["input"] == expected_images for request in requests) + assert image_prompt == _image_prompts()[0] + assert requests[0].extensions["timeout"] == {"connect": 7, "read": 7, "write": 7, "pool": 7} + + +@pytest.mark.parametrize("token", ["", " ", "bad token", "bad\n", "bad\u0080"]) +def test_invalid_responses_tokens_fail_before_dispatch(token: str, mock_client: MockClient): + def handler(request: httpx.Request) -> httpx.Response: + pytest.fail("An invalid token must not be sent.") + + mock_client(handler) + model = typechat.create_azure_openai_language_model( + None, "https://example.invalid/opaque", model="deployment", + token_provider=lambda: token, use_responses_api=True, + ) + assert _complete(model) == typechat.Failure( + "Token provider must return a nonempty, header-safe bearer-token string." + ) + + +def test_responses_token_provider_cancellation_prevents_dispatch(mock_client: MockClient): + async def provider() -> str: + raise asyncio.CancelledError() + + def handler(request: httpx.Request) -> httpx.Response: + pytest.fail("Cancelled token acquisition must not dispatch.") + + mock_client(handler) + model = typechat.create_azure_openai_language_model( + None, "https://example.invalid/opaque", model="deployment", + token_provider=provider, use_responses_api=True, + ) + with pytest.raises(asyncio.CancelledError): + _complete(model) + + +def test_responses_authentication_remains_mutually_exclusive(): + with pytest.raises(ValueError, match="not both"): + typechat.create_azure_openai_language_model( + "key", "https://example.invalid/opaque", model="deployment", + token_provider=lambda: "token", use_responses_api=True, + ) + with pytest.raises(ValueError, match="API key or a token provider"): + typechat.create_azure_openai_language_model( + None, "https://example.invalid/opaque", model="deployment", use_responses_api=True, + ) + + +@pytest.mark.parametrize("retry_limit", [0, 2, 3]) +def test_responses_exhausts_only_the_existing_retry_allowance(retry_limit: int, mock_client: MockClient): + requests: list[httpx.Request] = [] + + def handler(request: httpx.Request) -> httpx.Response: + requests.append(request) + return httpx.Response(503) + + mock_client(handler) + model = typechat.create_openai_language_model("key", "model", "https://example.invalid/opaque", use_responses_api=True) + model.max_retry_attempts = retry_limit + model.retry_pause_seconds = 0 + assert _complete(model) == typechat.Failure("Responses API request failed with HTTP status 503.") + assert len(requests) == retry_limit + 1 + + +@pytest.mark.parametrize("status", [400, 401, 403, 404, 405, 422, 429, 500, 502, 503, 504]) +def test_http_errors_never_switch_apis_or_expose_provider_payloads(status: int, mock_client: MockClient): + requests: list[httpx.Request] = [] + + def handler(request: httpx.Request) -> httpx.Response: + requests.append(request) + return httpx.Response(status, json={"error": {"message": "credential-or-prompt-marker"}}) + + mock_client(handler) + model = typechat.create_openai_language_model( + "key", "model", "https://example.invalid/responses", use_responses_api=True, + ) + model.max_retry_attempts = 1 + model.retry_pause_seconds = 0 + assert _complete(model) == typechat.Failure(f"Responses API request failed with HTTP status {status}.") + assert len(requests) == (2 if status in [429, 500, 502, 503, 504] else 1) + assert all("input" in json.loads(request.content) and "messages" not in json.loads(request.content) for request in requests) + + +@pytest.mark.parametrize("raw", [b"not JSON: private-marker", b"\xff", b'{"output":']) +def test_http_invalid_json_is_a_fixed_failure_without_a_retry(raw: bytes, mock_client: MockClient): + requests: list[httpx.Request] = [] + + def handler(request: httpx.Request) -> httpx.Response: + requests.append(request) + return httpx.Response(200, content=raw) + + mock_client(handler) + model = typechat.create_openai_language_model("key", "model", "https://example.invalid/responses", use_responses_api=True) + assert _complete(model) == typechat.Failure("Responses API returned malformed JSON.") + assert len(requests) == 1 + + +@pytest.mark.parametrize("advertised", [False, True]) +@pytest.mark.parametrize("oversized", [False, True]) +def test_responses_size_limit_accepts_b_bytes_and_rejects_b_plus_one( + advertised: bool, oversized: bool, mock_client: MockClient, +): + body = json.dumps(_response()).encode() + + async def chunks() -> AsyncIterator[bytes]: + yield body[:10] + yield body[10:] + + def handler(request: httpx.Request) -> httpx.Response: + return httpx.Response(200, content=body if advertised else chunks()) + + mock_client(handler) + model = typechat.create_openai_language_model("key", "model", "https://example.invalid/responses", use_responses_api=True) + model.max_response_bytes = len(body) - int(oversized) + result = _complete(model) + if oversized: + assert isinstance(result, typechat.Failure) + assert "maximum allowed size" in result.message + else: + assert result == typechat.Success("Hello!") + + +@pytest.mark.parametrize("limit", [0, -1]) +def test_nonpositive_response_size_limits_remain_disabled(limit: int, mock_client: MockClient): + mock_client(lambda request: httpx.Response(200, json=_response())) + model = typechat.create_openai_language_model("key", "model", "https://example.invalid/responses", use_responses_api=True) + model.max_response_bytes = limit + assert _complete(model) == typechat.Success("Hello!") + + +@pytest.mark.parametrize("cancelled", [False, True]) +def test_http_transport_failure_and_cancellation_preserve_request_bounds(cancelled: bool, mock_client: MockClient): + requests: list[httpx.Request] = [] + + def handler(request: httpx.Request) -> httpx.Response: + requests.append(request) + if cancelled: + raise asyncio.CancelledError() + raise httpx.ReadTimeout("private-address-or-prompt-marker", request=request) + + mock_client(handler) + model = typechat.create_openai_language_model("key", "model", "https://example.invalid/responses", use_responses_api=True) + model.max_retry_attempts = 0 + if cancelled: + with pytest.raises(asyncio.CancelledError): + _complete(model) + else: + assert _complete(model) == typechat.Failure("Responses API request failed.") + assert len(requests) == 1 + + +@dataclass +class _Value: + value: str + + +@pytest.mark.parametrize("use_client", [False, True]) +@pytest.mark.parametrize("with_images", [False, True]) +def test_translator_accepts_valid_responses_without_repair( + use_client: bool, with_images: bool, mock_client: MockClient, +): + model, requests = _model([_response('{"value": "validated"}')], use_client, mock_client) + translator = typechat.TypeChatJsonTranslator(model, typechat.TypeChatValidator(_Value), _Value) + preamble, expected = _image_prompts() if with_images else ([], []) + original = deepcopy(preamble) + + async def run() -> typechat.Result[_Value]: + async with _open_model(model): + return await translator.translate("get value", prompt_preamble=preamble) + + assert asyncio.run(run()) == typechat.Success(_Value("validated")) + assert len(requests) == 1 + assert requests[0]["input"][:-1] == expected + assert preamble == original + + +@pytest.mark.parametrize("use_client", [False, True]) +@pytest.mark.parametrize("with_images", [False, True]) +@pytest.mark.parametrize("initial", ['{"wrong": "field"}', '{"value":', "not JSON"]) +def test_translator_validates_and_repairs_responses_without_changing_mode( + initial: str, use_client: bool, with_images: bool, mock_client: MockClient, +): + model, requests = _model([_response(initial), _response('{"value": "repaired"}')], use_client, mock_client) + translator = typechat.TypeChatJsonTranslator(model, typechat.TypeChatValidator(_Value), _Value) + preamble: list[typechat.PromptSection] = [{"role": "system", "content": "context"}] + expected: list[_InputSection] = [{"role": "system", "content": "context"}] + if with_images: + preamble, expected = _image_prompts() + original = deepcopy(preamble) + + async def run() -> None: + async with _open_model(model): + assert await translator.translate("get value", prompt_preamble=preamble) == typechat.Success(_Value("repaired")) + + asyncio.run(run()) + assert len(requests) == 2 + assert requests[0]["input"][:-1] == expected + assert requests[0]["input"][-1]["role"] == "user" + assert requests[1]["input"][:-2] == requests[0]["input"] + assert requests[1]["input"][-2] == {"role": "assistant", "content": initial} + assert requests[1]["input"][-1]["role"] == "user" + assert preamble == original + + +@pytest.mark.parametrize("use_client", [False, True]) +@pytest.mark.parametrize("with_images", [False, True]) +def test_translator_preserves_final_failure_values_and_one_repair_default( + use_client: bool, with_images: bool, mock_client: MockClient, +): + model, requests = _model([_response('{"value": 123}'), _response("not JSON")], use_client, mock_client) + translator = typechat.TypeChatJsonTranslator(model, typechat.TypeChatValidator(_Value), _Value) + preamble, expected = _image_prompts() if with_images else ([], []) + original = deepcopy(preamble) + + async def run() -> typechat.Result[_Value]: + async with _open_model(model): + return await translator.translate("get value", prompt_preamble=preamble) + + result = asyncio.run(run()) + assert isinstance(result, typechat.Failure) + assert result.has_value and result.value == {"value": 123} + assert result.response_text == "not JSON" + assert len(requests) == 2 + assert all(request["input"][:len(expected)] == expected for request in requests) + assert preamble == original + + +@pytest.mark.parametrize("use_client", [False, True]) +@pytest.mark.parametrize("after_repair", [False, True]) +@pytest.mark.parametrize("with_images", [False, True]) +def test_model_output_failure_stops_translation_without_an_additional_repair( + use_client: bool, after_repair: bool, with_images: bool, mock_client: MockClient, +): + outputs = [_response('{"value": 123}')] if after_repair else [] + outputs.append({**_response(), "status": "incomplete"}) + model, requests = _model(outputs, use_client, mock_client) + translator = typechat.TypeChatJsonTranslator(model, typechat.TypeChatValidator(_Value), _Value) + preamble, expected = _image_prompts() if with_images else ([], []) + original = deepcopy(preamble) + + async def run() -> typechat.Result[_Value]: + async with _open_model(model): + return await translator.translate("get value", prompt_preamble=preamble) + + result = asyncio.run(run()) + assert isinstance(result, typechat.Failure) + assert len(requests) == (2 if after_repair else 1) + assert all(request["input"][:len(expected)] == expected for request in requests) + assert preamble == original + if after_repair: + assert result.has_value and result.value == {"value": 123} + assert result.response_text == '{"value": 123}' + else: + assert not result.has_value and result.response_text is None + + +@pytest.mark.parametrize("use_client", [False, True]) +def test_responses_rejects_malformed_image_preambles_without_repair( + use_client: bool, mock_client: MockClient, monkeypatch: pytest.MonkeyPatch, +): + provider = Mock(return_value="test-token") + model, requests = _model([], use_client, mock_client, token_provider=provider) + complete = AsyncMock(wraps=model.complete) + monkeypatch.setattr(model, "complete", complete) + translator = typechat.TypeChatJsonTranslator(model, typechat.TypeChatValidator(_Value), _Value) + preamble = cast(list[typechat.PromptSection], [ + {"role": "user", "content": [{"type": "image_url", "image_url": {"url": ""}}]}, + ]) + + async def run() -> typechat.Result[_Value]: + async with _open_model(model): + return await translator.translate("get value", prompt_preamble=preamble) + + result = asyncio.run(run()) + assert isinstance(result, typechat.Failure) + assert "section 0, part 0" in result.message + assert not result.has_value and result.response_text is None + assert requests == [] + provider.assert_not_called() + complete.assert_awaited_once() + + +def test_responses_import_and_protocol_need_no_provider_sdk(): + subprocess.run([ + sys.executable, "-c", + "import sys; sys.modules['openai'] = None; sys.modules['azure'] = None; " + + "from typechat import ResponsesClient, create_openai_language_model_from_client; " + + "assert callable(create_openai_language_model_from_client)", + ], check=True) diff --git a/site/src/docs/python/basic-usage.md b/site/src/docs/python/basic-usage.md index 59c63b1c..23707a08 100644 --- a/site/src/docs/python/basic-usage.md +++ b/site/src/docs/python/basic-usage.md @@ -75,9 +75,10 @@ You can call these directly. def create_openai_language_model( api_key: str, model: str, - endpoint: str = "https://api.openai.com/v1/chat/completions", + endpoint: str | None = None, org: str = "", *, + use_responses_api: bool = False, temperature: float | None = 0.0, ): ... @@ -87,6 +88,8 @@ def create_azure_openai_language_model( endpoint: str, *, token_provider: TokenProvider | None = None, + use_responses_api: bool = False, + model: str | None = None, temperature: float | None = 0.0, ): ... ``` @@ -98,6 +101,8 @@ you're using OpenAI or Azure OpenAI. def create_language_model( vals: dict[str, str | None], *, + use_responses_api: bool = False, + model: str | None = None, temperature: float | None = 0.0, ) -> TypeChatLanguageModel: ... ``` @@ -183,17 +188,110 @@ Your client's endpoint, API version, authentication, retries, and timeouts remain configured by your application. TypeChat does not reconfigure or close the client, or add another retry/timeout layer. Client exceptions and cancellation propagate unchanged. Missing choices or text produce a `Failure`. -The adapter is for Chat Completions, not the Responses API. +These are the default Chat Completions settings. To use the client's Responses +API instead, select it explicitly as shown below. Pass either model to `TypeChatJsonTranslator` as usual. Schema generation, validation, and repair behavior remain unchanged. +#### Using the Responses API + +For new text and image integrations, select Responses explicitly with +`use_responses_api=True`. All existing calls still default to Chat Completions; +the OpenAI factory's omitted endpoint still resolves to +`https://api.openai.com/v1/chat/completions`. No API is inferred from a URL. + +The HTTP path requires the **complete Responses destination** and an explicit +model or deployment name: + +```py +from typechat import create_openai_language_model + +async with create_openai_language_model( + api_key, + model="my-model", + endpoint="https://api.openai.com/v1/responses", + use_responses_api=True, +) as model: + translator = TypeChatJsonTranslator(model, validator, target_type) + result = await translator.translate("Classify this request") +``` + +For Azure, use `create_azure_openai_language_model` with +`use_responses_api=True` and `model="my-deployment"`. Either an API key or the +existing `api_key=None, token_provider=token_provider` authentication works. +Supply the exact Responses URL supported by your Azure resource and API +version. OpenAI-compatible routes, Azure deployment-scoped/query-versioned +routes, and opaque proxy destinations are passed through without rewriting +paths, queries, or model names. Not every provider or Azure API version +supports Responses; TypeChat does not probe endpoint capabilities. + +`create_language_model(settings, use_responses_api=True)` uses the existing +`OPENAI_ENDPOINT`, `OPENAI_MODEL`, and API-key/organization settings. +For Azure, use the existing `AZURE_OPENAI_ENDPOINT` and API key and pass +`model="my-deployment"` explicitly. The optional `model` keyword also overrides +`OPENAI_MODEL`; no new environment-variable names are required. + +To reuse a configured asynchronous Responses client: + +```py +from typechat import create_openai_language_model_from_client + +model = create_openai_language_model_from_client( + openai_client, + model="my-model-or-azure-deployment", + use_responses_api=True, +) +translator = TypeChatJsonTranslator(model, validator, target_type) +result = await translator.translate("Classify this request") +``` + +The exported structural `ResponsesClient` protocol requires only +`client.responses.create`; a Chat-only client still needs only the unchanged +`ChatCompletionClient` contract in default mode. No SDK import or TypeChat +environment configuration is needed. The supplied instance owns authentication, +base URL, API version, transport limits, retries, and cleanup. Exceptions and +cancellation propagate unchanged; TypeChat adds no client retries or timeouts. + +Both paths send `input`, `model`, and `stream=False`, not Chat Completions' +`messages`, `temperature`, or `n`. String prompts and tagged text parts work in +system/user/assistant sections; image parts work only in user sections. Use the +same public `text` and `image_url` prompt types shown in +[text and image prompts](#text-and-image-prompts). Responses models map user/system +text to `input_text`, assistant text to `output_text`, and images to `input_image`, +flattening the image reference to `image_url` and preserving optional `detail`, +message/part order, and caller-owned data. They validate prompts before +authentication or dispatch. Tools, streaming, and remote conversation-state +management remain outside this integration. + +Responses must have a completed canonical envelope and completed assistant +messages. Ordered `output_text` parts are concatenated; known reasoning items +do not contribute text. An explicitly empty text string is allowed, but missing +or malformed content, refusals/blocking, incomplete/failed/nonterminal responses, +and unsupported output (including tool calls) return a fixed, payload-free +`Failure`, even alongside partial text. A client's convenience `output_text` +property cannot bypass these checks. A successful model completion still goes +through TypeChat's normal JSON/schema validation and bounded repair. + +HTTP models retain three retries, a one-second retry pause, a ten-second timeout, +and a 100 MiB response limit by default, including incremental response-size +checks. These controls remain configurable. Token providers run afresh on each +attempt; cancellation propagates. HTTP failures do not expose raw provider +payloads, and malformed Responses output does not cause a retry. + +Automatic Responses-to-Chat fallback is deferred: there is no switch, URL +inference, capability cache, or extra fallback request. Keep using default Chat +Completions mode for services that do not support Responses. + +See the [runnable Responses example](https://github.com/microsoft/TypeChat/blob/main/python/examples/responses/demo.py) +for a standalone HTTP call and a function accepting an application-owned client. + #### Configuring temperature All four model factories accept a keyword-only `temperature: float | None` -option. Leaving it unspecified retains the existing `0.0` setting. Pass a number -to configure it, or `None` to omit the field entirely (not send JSON `null`) and -use the configured client's or service's default: +option for Chat Completions. Leaving it unspecified retains the existing `0.0` +setting. Pass a number to configure it, or `None` to omit the field entirely +(not send JSON `null`) and use the configured client's or service's default: ```py model = create_azure_openai_language_model( @@ -214,6 +312,7 @@ Omission supports reasoning models that reject explicit sampling parameters, including models that do not support disabling reasoning. It does not set a reasoning effort, infer model capabilities, change authentication, or switch APIs. Other client settings, ownership, retries, and timeouts are unchanged. +Responses requests always omit temperature, regardless of this option. #### Using OpenAI-compatible gateways @@ -245,6 +344,7 @@ few writable attributes you might find useful: - `max_retry_attempts` - `retry_pause_seconds` - `timeout_seconds` +- `max_response_bytes` Though note that these are unstable. @@ -264,9 +364,10 @@ model = typechat.create_language_model(os.environ) ## Text and image prompts -The HTTP Chat Completions models and configured-client adapter accept image -references alongside text. Use `model.complete` for text output, or supply image -context through the translator's existing `prompt_preamble` for validated data. +The HTTP models and configured-client adapters accept image references alongside +text in both Chat Completions and explicitly selected Responses mode. Use +`model.complete` for text output, or supply image context through the translator's +existing `prompt_preamble` for validated data. The calls below demonstrate both interfaces; an application can use either independently. Replace the hosted URL with your own and pass already-base64-encoded image data: @@ -340,6 +441,13 @@ ordering, exact URL/data-URL strings, and caller-owned dictionaries. Malformed or unsupported input returns a `Failure` identifying the section or part without echoing its content, before resolving a token or dispatching a request. +These public types are shared by both APIs; callers do not construct Responses +wire-format parts. In Responses mode, tagged `text` becomes `input_text` for +user/system messages or `output_text` for assistant messages. Tagged `image_url` +becomes `input_image` with the original reference in a flat +`image_url` field. Supplied `detail` moves alongside it; omitted detail remains +omitted. Images are never flattened into text, including during translator repair. + The translator's current request remains a string, and ordinary `translate(str)` calls need no changes. Preamble messages remain before the combined schema/request message; validation and repair keep the original context and append rejected @@ -352,7 +460,7 @@ Supply a provider/model that supports images and follow its supported file forma (for example PNG, JPEG, or WebP), accessible-URL requirements, image-count limits, and encoded-payload, resolution, and size limits. TypeChat validates reference structure only. It never fetches, decodes, resizes, or validates image bytes and -does not add image generation, audio/video, streaming, or Responses API support. +does not add image generation, audio/video, or streaming. No new dependency is required. Existing API-key/token-provider authentication, HTTP retries/timeouts/response-size limits, and caller-owned SDK configuration and lifecycle remain unchanged. Custom models must handle the prompt parts they accept.