Repository navigation
Conversation
Byte-identical to the media_capture.py change on feat/attachments-python-auto (1b3072b + 0b2eb4b), so merging that branch later is a no-op here. Provider adapters read CAPTURE_POLICY_KEY / OWNER_KEY through local_capture_policy and Observation. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…LM and compatible chat New fi_instrumentation.attachments.provider_media (mirrors the TypeScript provider-media helper): preflight before any read, exact-type re-admission, magic-byte check after admission, deny-wins policy through media_capture, and a per-package capability table that is the Python support matrix. Bindings: google-genai (generate_content in/out, stream input), vertexai (GenerateContent in/out, Imagen predictions), bedrock (Converse incl. tool screenshots, InvokeModel Anthropic/Nova/Titan/Nova Canvas), litellm (completion input, image_generation output, owner claim around the downstream SDK call), groq/together/fireworks/xai/mistralai/openrouter/ portkey/ollama chat media. deepseek and cerebras export an empty descriptor (no qualified media surface). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…a; CI workflow Vertex Imagen predictions are a proto repeated field that does not slice, so the bounded walk uses islice. traceai_together installs no chat wrapper today, so its descriptor claims nothing. New workflow runs every provider binding at its declared floor and latest SDK with outbound network off. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The LiteLLM package suite calls the wrapper methods with a stand-in self, so the media helpers are module functions that read the manager with getattr (no manager: unchanged behavior). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A file plus a node ID in the same file collapsed to the node ID, so the compatible-chat jobs only ran the descriptor rows. Each job now also requires its minimum number of passed rows, and the step runs with pipefail. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ap is exact Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…wner, vLLM/MiniMax bindings, Vertex descriptor, MP4 brand, data-URI bound, InvokeModel part index) - LiteLLM claims the call from a nested OpenAI instrumentor, so it now captures PDF file/input_file data URIs itself (documents=True, LiteLLM only): one upload. - traceai_xai defers to an enabled traceai_openai like fireworks: one owner per call. - Thin vLLM (OpenAI dialect) and MiniMax (image dialect) bindings with descriptors; MiniMax skips media while traceai_openai is enabled. - The vertexai descriptor advertises the proven PredictionServiceClient paths only. - sniff() accepts video/mp4 only for MP4 ISO-BMFF major brands (not HEIC/AVIF/QT/M4A). - Data URIs go through media_capture.split_inline: bounded before slicing. - Bedrock InvokeModel tool_result children carry the block index, as Converse does. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ip LiteLLM releases that reject file parts The span ref carries message and part indexes only. litellm 1.53.x rejects chat file parts before sending, so the floor job proves the PDF row on releases that accept them (latest) and keeps its 9-row minimum. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…r message) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…local deny traceai_xai and traceai_minimax now hand the whole chat call to an enabled traceai_openai only while attachments are enabled for them. With attachments off they trace exactly as before (binding span plus the OpenAI span). A deferred call runs under provider_media.deferred(config): the binding's local hide/PII flags are folded into a restriction-only CapturePolicy and merged (deny wins) with the per-call policy under CAPTURE_POLICY_KEY, so the OpenAI instrumentor cannot capture more than the binding allows, whatever its own TraceConfig. hide_input_images denies every input part of the deferred call (a CapturePolicy cannot single out images). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
nik13
added a commit
that referenced
this pull request
Nov 12, 2026
Cherry-picked from e158682 (feat/attachments-python-providers, #697). Every non-docstring hunk of media_capture.py is byte-identical to that commit (and to feat/attachments-python-auto), so merging those branches later is a no-op here. The docstring keeps both bullets; the next commit rewrites the per-span one. (cherry picked from commit e158682) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ptures traceai_xai and traceai_minimax handed the whole call to traceai_openai whenever it was enabled. In a mixed setup (binding with an attachment manager, OpenAIInstrumentor enabled without one) that dropped the binding span and its media: nothing was uploaded. OpenAIInstrumentor gains a small public accessor, captures_attachments(), true only while it is instrumented with an active shared manager. The bindings now defer only when both sides capture; otherwise they keep the call, trace it and capture its media themselves. An older traceai_openai without the accessor is treated as not capturing. Tests: mixed setup in both orders (sync and async) uploads each image once, owned by the binding span, with both spans kept; legacy-accessor stub never takes the call. CI minimums: xai 15, minimax 16. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… has no manager) With traceai_openai capturing and xAI/MiniMax instrumented without a manager, the binding keeps its own span (tracing unchanged) and the image is uploaded once, owned by the OpenAI span. CI minimums: xai 17, minimax 18. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Part of TH-2379 (Linear).
Summary
Python provider media for Gemini, Vertex, Bedrock, LiteLLM and the OpenAI-compatible chat packages (work packages M05, M06, M08, M10 and M12). It mirrors the TypeScript
@traceai/provider-mediahelper (#683): one shared helper, thin per-package bindings, and a capability table that is also the support matrix.Stacked on #690 (
feat/attachments-python-openai-anthropic). It reuses that PR'smedia_capturehelper; it does not fork it.fi_instrumentation.attachments.provider_media(adapter-facing, not inattachments.__all__):ProviderMedia(manager, span, namespace, trace_config=)has typed walkers:gemini_contents/gemini_response,compatible_messages(messages, dialect),ollama_messages/ollama_images,generated_images,bedrock_converse,anthropic_messages,bedrock_invoke_request/bedrock_invoke_responseandimagen_predictions.manager_from_kwargs,claim(context),sniff,consistentandcapabilities(package).PROVIDER_MEDIA_CAPABILITIESis the support matrix.media_capture.pyhunks fromfeat/attachments-python-auto(1b3072b + 0b2eb4b) unchanged, byte for byte. That bringsCAPTURE_POLICY_KEY,OWNER_KEY,owned_elsewhereandmerge_capture_policies, so merging that branch later changes nothing here. Every walker goes throughmedia_capture.local_capture_policyandObservation. As a result the per-call policy key and the owner key apply to provider media, and no binding writes either key as a string literal, except the deny-all fallback intraceai_xai/traceai_minimax, used only ifprovider_media.deferredcannot be loaded.attachments=(and auto'sattachment_manager=). Each package exports<name>_attachment_capabilitieslazily.Support matrix
traceai_google_genaimodels.generate_content: inputinline_dataParts (image, audio, video, PDF) and output candidateinline_data(generated images, audio, files).generate_content_stream: input onlyfile_data/gs://URIs, provider files, PIL images, streamed output,generate_images,generate_videos, Live,audio/L16PCMtraceai_vertexaiPredictionServiceClientonly:generate_content(v1, v1beta1) input and outputinline_data, and Imagenpredictoutput (bytesBase64Encodedwith a declared image type, only on/models/imagen-*/imagegenerationendpoints)file_data/gs://, streamed output, the async client, any other prediction endpoint.GenerativeModelandImageGenerationModelrun through the same client but have no fixture yet, so the descriptor does not claim themtraceai_bedrockimage/documentbyte blocks for Claude 3/4 and Nova lite/pro/premier. OnetoolResultlevel: a computer-use screenshot is recognized only when a typedcomputer_YYYYMMDDtool is declared AND an assistanttoolUseof that tool has the ID. Novavideobytes. InvokeModel: Anthropic Messages bodies, Nova bodies, Titan Image Generator / Nova Canvas input images and outputimages, read from the body the instrumentor already bufferedtraceai_litellmcompletion/acompletion/completion_with_retrieschatimage_urldata URIs,input_audio, and PDFfile(file.file_data) /input_file(file_data) data URIs.image_generation/aimage_generationb64_jsonoutputapplication/octet-streamfiles, streamed output, image edit, speech, transcriptiontraceai_groq(chat)chat.completions.createimage_urldata URIs (image only, as in TypeScript)input_audio, remote URLs, output media. Speech and transcription stay as in #690traceai_fireworksimage_urldata URIs. It defers the whole call totraceai_openaiwhile that is enabled (pre-existing rule)traceai_xaiimage_urldata URIs. With attachments enabled for it andtraceai_openaienabled with an active attachment manager, it hands the whole call totraceai_openaiunder its own local deny; with attachments off it traces exactly as beforetraceai_vllm(legacyVLLMInstrumentor)image_urldata URIs andinput_audio(wav/mp3). It already defers totraceai_openaiwhile that is enabledvideo_url, output mediatraceai_minimaximage_urldata URIs (MiniMax-M3 / M3.1-Flash accept images; there is no audio input). With attachments enabled for it andtraceai_openaienabled with an active attachment manager, it hands the whole call totraceai_openaiunder its own local deny; with attachments off it traces exactly as beforevideo_url, audio, output media, native speech/image/video (X04)traceai_mistralaichat.complete/stream(and async)image_url(string or object) data URIs, in both the 1.x wrapper path and the 3.6+ native-hook pathdocument_url,input_audio, remote URLstraceai_openrouterchat.sendimage_urldata URIs andinput_audio(wav/mp3)traceai_portkeyimage_urldata URIs andinput_audiotraceai_ollamachatmessages[].imagesandgenerateimages: bytes or base64, typed by magic bytestraceai_togethertraceai_togetherinstalls no chat wrapper today: its install check looks forchaton theTogetherclass, where it is an instance attribute. The binding is wired, but no media is claimed until that pre-existing bug is fixedtraceai_deepseek,traceai_cerebrasPrivacy defaults (unchanged)
manager.preflightruns before a source is read, decoded or sniffed.audio/wavis not uploaded, so a wrong label cannot get pasthide_input_images.TraceConfighide/PII flags, the per-call context policy and the owner mark all apply.gs://, S3, provider file or filesystem path is ever fetched or opened.media_capture.OWNER_KEY, so a nested OpenAI instrumentor uploads nothing more. LiteLLM therefore captures every chat part kind the OpenAI instrumentor would have captured there (images, PDF files) plusinput_audio. No other binding claims. Every observing call owns its own occurrence; there is no cross-call claim.traceai_xai,traceai_minimax,traceai_fireworksandtraceai_vllmwrap the sameopenaimethod astraceai_openai.traceai_fireworksandtraceai_vllmdefer the whole call to it while it is enabled (their existing rule).traceai_xaiandtraceai_minimaxdefer the whole call only while attachments are enabled for them ANDtraceai_openaiholds an active attachment manager (OpenAIInstrumentor.captures_attachments()); with attachments off they keep both spans, exactly as on base, and next to an OpenAI instrumentor without a manager they trace and capture the call themselves. Either setup order works, and each part is uploaded once.provider_media.deferred(config): the binding's localTraceConfigbecomes a restriction-onlyCapturePolicy(pii_redactiondenies all; an input hide flag orhide_input_imagesdenies inputs; an output hide flag denies outputs), merged deny-wins with any per-call policy undermedia_capture.CAPTURE_POLICY_KEY.traceai_openaitherefore cannot capture more than the binding allows, whatever its ownTraceConfig. ACapturePolicycannot single out images, sohide_input_imageson the binding also denies input files on the deferred call.video/mp4needs an MP4 ISO-BMFF major brand (isom,iso2–iso6,mp41,mp42,avc1,dash,M4V,mmp4). A HEIC/AVIF still image, QuickTime or M4A labelledvideo/mp4is refused. Data URIs go throughmedia_capture.split_inline, which bounds the length before it slices.Notes for other lanes
<name>_attachment_capabilitiesfor_MEDIA_ADAPTERS; this now includestraceai_vllmandtraceai_minimax. Leave outtraceai_deepseek,traceai_cerebrasandtraceai_together: their descriptors have no operations.traceai_xai/traceai_minimaxandtraceai_openaienabled, a call has one span (the OpenAI instrumentor's) instead of two. With attachments off, both spans stay in either setup order (CI rowtest_binding_without_attachments_keeps_its_span); a canonical span diff against base is still owed.attach_bytestomedia_capture. This PR keeps its bytes path insideprovider_media, so the two do not conflict textually. The reconciliation should keep one.Review fix pass (head 774675b)
Fixes from the independent review of 73727e7:
file/input_filePDF parts the nested OpenAI instrumentor would otherwise capture. LiteLLM now captures them itself (compatible_messages(..., documents=True), LiteLLM only). Reviewer probe: base 1 upload, previous head 0, now 1 (completion,document).traceai_xaidefers to an enabledtraceai_openai, as Fireworks does. Reviewer probe in both setup orders: previous head 2 uploads, now 1 (ChatCompletion).traceai_openaiis enabled; a test proves one upload in both orders.PredictionServiceClientpaths;GenerativeModel/ImageGenerationModelare listed as unsupported until they have fixtures.sniff. Checks the ISO-BMFF major brand: HEIC/AVIF/QuickTime/M4A labelledvideo/mp4is refused.media_capture.split_inline, so the length is bounded before any slice.tool_result. Children carry the block indexp, as Converse does.Deferral gating and deny propagation (head d058417)
Binding decisions applied to
traceai_xaiandtraceai_minimax:traceai_openaionly while attachments are enabled for the binding (attachments=/attachment_manager=resolved to an active manager). With attachments off, the code path is base's: the binding span and the OpenAI span both stay. This restores base tracing for xAI (774675b had dropped the xAI span even with attachments off). MiniMax now defers the whole call instead of keeping its span and skipping media.provider_media.deferred(self._config)(new, see Privacy defaults). A binding withhide_input_images, an input hide flag orpii_redactiontherefore uploads nothing, even whentraceai_openaion the same manager uses a defaultTraceConfig. Reviewer probelocal-deny-openai-defaulton 774675b: payloads=1.New rows:
test_binding_without_attachments_keeps_its_span(both orders, sync + async),test_deferred_call_keeps_binding_local_deny(no flag /hide_input_images/hide_inputs/pii_redaction, sync + async),test_deferred_call_keeps_context_policy_and_owner, plus core unit rows fordeferral_policy/deferred. Job minimums were raised: core 54, xai 12, minimax 13.CI on d058417: 48/48 checks pass. Passed rows: core 54, xai 12 (floor and latest), minimax 13 (floor and latest).
The private-VM qualification owed here was run on head 11e6b91 (next section).
Mixed setup fix and VM proof (head 11e6b91)
Review P2:
_defers()only checked thattraceai_openaiwas enabled. With the binding holding a manager andOpenAIInstrumentorenabled without one, the call was handed to an instrumentor that captures nothing: the binding span and its media were both lost (0 uploads).OpenAIInstrumentor.captures_attachments()is true only while it is instrumented with an active shared manager (the same_attachment_managerresolution it already used). It imports nothing.traceai_xai/traceai_minimaxdefer only when they have a manager ANDtraceai_openaicaptures. Otherwise they keep the call, trace it and capture its media themselves. An oldertraceai_openaiwithout the accessor counts as not capturing.test_binding_keeps_call_when_openai_has_no_manager(both orders, sync + async: one upload per call, owned by the binding span, both spans kept);test_binding_without_manager_keeps_its_span_next_to_capturing_openai(both orders: binding span kept, one upload owned byChatCompletion);test_older_openai_without_accessor_never_takes_the_call. Job minimums: xai 17, minimax 18.CI on 11e6b91: all checks pass; the new rows ran (xai 17 floor/latest, minimax 18 floor/latest, core 54).
Private VM on 11e6b91 (synthetic, loopback only, outbound network disabled):
local-deny-openai-default: payloads 1 -> 0 (xai and minimax);openai-no-manager: payloads 0 -> 1, owned by the binding span, both spans kept (the P2 fix);[ChatCompletion]->[ChatCompletion, xAI Chat Completions](base tracing restored).hide_input_imagesnot folded, core and compat;pii_redactionnot folded; deferral back to "OpenAI enabled" for xai and minimax; accessor ignoring the manager).traceai_minimax/_wrappers.py, all present on base.Testing
All runs are synthetic. The real installed SDKs talk to a loopback provider server and a TLS loopback F01 attachment server, with outbound network disabled (
unshare --net). No real-provider call was made; real-provider evidence (Q04) is still owed.Fix pass, private VM, every CI row reproduced with the workflow's exact install and test mapping (floor on Python 3.10, latest on 3.12). 31 rows ran on b68e065; the commits after it change tests only, and the 4 affected rows (bedrock and litellm, floor and latest) were re-run on 774675b. CI on 774675b is the full run of the final head. All 33 rows pass at or above their minimum passed counts: core 45, google-genai 9, vertexai 5, bedrock 12, litellm 9 floor / 10 latest, groq 2, together 3, fireworks 3, xai 5, vllm 4, minimax 6, deepseek 3, cerebras 3, mistralai 5, openrouter 3, portkey 3, ollama 4.
fileparts before sending, so the PDF row skips there (the floor minimum stays 9) and runs on 1.109.1.part_indexback toq; LiteLLMdocuments=Truedropped; file parts not opt-in; xAI deferral removed; MiniMax OpenAI skip removed; vLLM capture call removed; MiniMax localTraceConfigignored.CI (earlier run of record: head 73727e7). All 44 checks pass, including the 29 new "Provider media" jobs and the #690 "Provider attachments" floor/latest jobs.
Private VM, same suites at floor (Python 3.10) and latest (Python 3.12):
httpx<0.28and green in CI.tests/attachments_providers) passes 65/65 on this head.Regression check: each touched package's own test suite, run on head and on base (f5e77e0):
self, which gave 45 failures. It was fixed in d0ea4e8 (now 201/201 on both tiers). CI has no LiteLLM package workflow.Mutations: 19/19 detected. Each mutation was applied to the final tree and run against the latest venvs; the mistralai 1.x wrapper mutation used the 1.1.0 venv.
Owed:
GenerativeModel/ImageGenerationModelfixtures (they go through the same gapic wrap, but the descriptor does not claim them until tested), the async client and streamed output.generate_images/generate_videos(not wrapped).🤖 Generated with Claude Code