fix: treat http/https URL strings as image_url in multimodal messages - #8192
Open
MOHAMMED WASIM KHAN (wasim-builds) wants to merge 9 commits into
Open
fix: treat http/https URL strings as image_url in multimodal messages#8192MOHAMMED WASIM KHAN (wasim-builds) wants to merge 9 commits into
MOHAMMED WASIM KHAN (wasim-builds) wants to merge 9 commits into
Conversation
added 9 commits
September 4, 2026 12:15
Added a new cancellation.md page documenting how CancellationToken propagates through teams, agents, workbenches, and tools. Includes a runnable example of a cancellable nested tool and a checklist of call sites that must forward the token. Fixes microsoft#8091
Cache key was built from messages, tools, json_output, and extra_create_args only. Calls that differed only in tool_choice (e.g. auto vs none) collided and returned stale cached responses. Added tool_choice to the cache key data. For Tool objects, the serialized key uses the tool name; for string choices, the string is used directly. Fixes microsoft#7968
…string Previously, return_value_as_string fell back to str() for dict/list values, producing Python repr strings instead of valid JSON. This broke downstream agents that expected parseable structured data. Now dict and list return values are serialized with json.dumps(), matching the existing BaseModel path. Other types still use str() as a safe fallback. Fixes microsoft#7867
Added GEN_AI_AGENT_ACTION_REF constant and derive_action_ref helper for deterministic cross-producer audit correlation. Added optional action_ref parameter to trace_tool_span(). Fixes microsoft#7850
…nClient Streaming tool_call chunks can have None fields for id, function.name, or function.arguments. Added None checks before concatenation to prevent TypeError during streaming aggregation. Fixes microsoft#7157
Added http_client field to both the TypedDict and Pydantic config model so it can be passed through component config dicts. The field is excluded from serialization via Field(exclude=True) since httpx.AsyncClient instances are not JSON-serializable. Fixes microsoft#7107
HttpToolConfig fields 'description' and 'headers' were Optional without defaults, so model_dump(exclude_none=True) omitted them and _from_config() raised ValidationError when reconstructing the tool. Added = None defaults to both fields so round-tripping through dump_component/load_component works correctly. Fixes microsoft#7172
The MCP Python SDK 2.x exposes snake_case attributes (input_schema, is_error) on Tool and CallToolResult, but autogen-ext was accessing camelCase aliases (inputSchema, isError) which raise AttributeError in newer SDK versions. Changed all occurrences in _base.py and _workbench.py. Fixes microsoft#7129
When a MultiModalMessage contains a string starting with http:// or https://, the OpenAI message transformer now emits an image_url content part instead of text. This allows passing remote image URLs directly to multimodal models without requiring an Image wrapper. Fixes microsoft#7005
Sylvester Kaczmarek (sylvesterkaczmarek)
left a comment
Sylvester Kaczmarek (sylvesterkaczmarek)
left a comment
There was a problem hiding this comment.
A URL string is not necessarily an image. With this change, ordinary text such as https://example.com/docs inside MultiModalMessage.content is silently retyped as image_url, changing message semantics and potentially causing provider errors. Could remote-image conversion be explicit (or at least validate that the URL is actually intended as image content) rather than keying only on the scheme?
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What happened
MultiModalMessagetreated all strings as text, even when they were remote image URLs likehttps://example.com/image.png. Multimodal models expect those to be sent asimage_urlcontent parts, not as text.Fix
In
_set_multimodal_content, strings that start withhttp://orhttps://are now emitted as:{"type": "image_url", "image_url": {"url": "https://..."}}instead of being wrapped as text. This matches the OpenAI vision API format and lets users pass remote image URLs directly in
MultiModalMessage.content.Testing
No new tests. I manually verified by constructing a
MultiModalMessagewith a remote URL and confirming the transformed OpenAI message contains animage_urlpart.Fixes #7005