Repository navigation
feat(python): framework media capture, one owner per model call (LangChain, LlamaIndex, Agno, Pydantic AI) - #691
Draft
nik13 wants to merge 17 commits into
Conversation
LangChain and LlamaIndex message content blocks, Agno Image/Audio/File and Pydantic AI BinaryContent reach the shared attachment manager. The shared helper gains a per-trace claim table, so a framework span and the provider span beneath it (traceai_openai, traceai_anthropic) upload identical bytes once. A later observer must pass its own policy before reusing a claim, and only admitted bytes are hashed. URLs go to the core RemoteUrl policy and are never fetched; paths, streams, provider files and unknown objects are never read. Agno capture uses the intersection of the FI and OpenInference TraceConfig. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…c-ai 0.1.0 needs Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
LangChain and LlamaIndex do not make their spans current, so the provider span beneath them starts a separate trace. Claims are now keyed by manager, MIME and digest with a lazy 300 s expiry, so the framework and provider observations of one call still upload once. Tests cover separate traces, another manager and claim expiry; pydantic-ai tests skip the dependency check because CI installs pydantic-ai-slim. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Patching time.monotonic globally also moved the manager's own deadlines. The floor tier pins an OpenTelemetry SDK that pydantic-ai 0.1.0 can import. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… and installs fastapi pydantic-ai 0.1.0 imports opentelemetry._events, which API 1.39+ no longer ships, and Agno imports fastapi from its workflow module. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ent results pydantic-ai 0.1.0 hangs in a forked child with no instrumentation at all, so the fork exit probe runs on LangChain only. Its FunctionModel also cannot serialize a binary tool return on the second request; the tool span has captured the image by then, which the test still asserts. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…i 0.x Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…h hide directions Remove the per-manager claim table from media_capture. It reused one occurrence across spans and traces, so a later span carried a fi-attachment URI that its own fi.attachment.refs did not list and that F01 resolve (which checks the expected parent) could never resolve. Every observing span now uploads and references its own occurrence; byte deduplication is left to the server. This also removes the dropped-upload reuse and the claim lock held across fork. capture_parts gains policy_directions. The Pydantic AI tool span folds both the input and the output hide flags (deny-wins) for the media a tool returns, because that span redacts its result under the output flags and the result is also a later model input (as in the TS MCP binding). Tests: claim tests inverted (two spans, two occurrences, each referenced on its own span); provider-only two-call regression; a fi-attachment URI resolution invariant on every real-provider test; hide_outputs / hide_output_text / hide_output_messages negative controls on the tool span. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…orkflow comment Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…currence tests capture_parts handed attach_bytes the bytes of a "bytes" part before any preflight, so a part denied by a local hide flag was still read. attach_bytes now also takes a zero-argument callable and calls it only after a preflight at the part cap passed every privacy, correlation and sampling step (too_large / queue_full still size exactly, as in attach_inline). Tests: span refs carry purpose and indices, not tool_call_id (that is in the upload request), and a provider span projects its fi-attachment URI into several attributes, so the provider-only test now checks that every URI resolves to its own span and each span carries its own. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Upload workers run concurrently, so the two parents can reach the loopback F01 server in either order (seen once in 15 runs on a VM). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This was referenced Nov 12, 2026
Cherry-picked from e158682 (feat/attachments-python-providers, #697). Every non-docstring hunk of media_capture.py is byte-identical to that commit (and to feat/attachments-python-auto), so merging those branches later is a no-op here. The docstring keeps both bullets; the next commit rewrites the per-span one. (cherry picked from commit e158682) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
When the LangChain, LlamaIndex, Agno or Pydantic AI adapter captures media for a model call (attachments enabled), the downstream provider SDK call now runs under the owner mark (media_capture.OWNER_KEY), so traceai_openai / traceai_anthropic beneath it upload nothing for that call. Framework + OpenAI on one manager goes from 2 uploads per part to 1, held by the framework span. - core: CallOwner, call_owner (gated on an active manager and an unowned call), owner_scope / owned_call, and owned_stream, which marks each advance of a returned (async) generator and never the caller's code between chunks. capture_parts treats a CallOwner of its own namespace as its own, so the LangChain and LlamaIndex callbacks still capture inside the mark. - LangChain: BaseChatModel generate/agenerate/stream/astream marked (patched only while attachments are enabled; restored on uninstrument). - LlamaIndex: chat/achat/stream_chat/astream_chat of every LLM class (patched like StreamInstrumentation, including later-imported providers). - Agno: the owner mark joins span_scope's values (call and each stream advance). - Pydantic AI: request marked; request_stream only while entered and exited, never the caller's async-with body. - DIFFERENT calls still each own their occurrence (same bytes twice = 2). - Attachments off: nothing marked or patched; tracing is the base path. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…arking it LlamaIndex's async stream proxy also defines send/throw/close, so owned_stream checked it as a sync generator first (CI: llamaindex-astream TypeError). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…es its media 16311a3 set the owner mark on every framework model call while attachments were enabled. Media the adapter deliberately skips was then captured by nobody: an Agno Image/File(filepath=...) or a LlamaIndex ImageBlock/DocumentBlock(path=...) is inlined as a data URI by the SDK integration, but traceai_openai skipped the marked call. Framework + traceai_openai on one manager (the auto default) went from 1 upload (provider span) to 0. - Agno, LangChain, LlamaIndex and Pydantic AI now mark a call only when the adapter's own parts scan yields at least one input media part (observes_agno_input / observes_langchain_input / observes_llamaindex_messages / PydanticAIMedia.observes_request). Every gate fails open: an error means no mark. - The gate is on OBSERVED parts, not admitted ones: a part the framework hides (hide_inputs, pii_redaction) is still observed, so the mark keeps carrying the framework's deny to the provider (test_framework_hide_inputs_... stays 0). - Regression rows (framework + traceai_openai): Agno filepath image/file and LlamaIndex path image/document -> 1 upload, on the provider span; framework alone -> 0 (negative control); a call mixing a content part and a path part -> 1 upload on the framework span (disclosed residual: the path part is skipped). - Probe rows: the provider call beneath LangChain/LlamaIndex is unmarked for text, provider-file and path-only calls, and marked for an observed image. - media_capture.py: docstring only. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
LangChain ChatAnthropic + traceai_anthropic on one manager (sync and stream): one upload, on the framework span; with framework attachments off the Anthropic span owns it. Latest tier only; CI installs traceai_anthropic, langchain-anthropic 1.7.10 and anthropic 2.8.0 (floor pins anthropic 0.41.0). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…gate VM mutation run on a2a007f left two survivors: - dropping the owner check from Observation alone (local_capture_policy still sets requested=False, so no row changed); each check now has its own assertion; - Pydantic AI _call_owner marking a request it does not observe; now asserted directly (text-only request -> no mark). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Part of TH-2379 (Linear).
What this does
Framework instrumentors now capture typed media through the shared attachment manager. One model call, one owner: when a framework adapter observes a model call's input media, the provider instrumentor beneath it skips that call, so each part is uploaded once. A call whose media the adapter never reads (an Agno
filepath, a LlamaIndexpath) is not marked, so the provider span still captures what the SDK inlines. Different calls each get their own occurrence.dumpdform the tracer receives): OpenAI-styleimage_url/file/input_audiodata, LangChain 0.3source_type: "base64"blocks, LangChain 1.xbase64content blocks and Anthropic-stylesourceblocks.ToolMessagemedia keeps itstool_call_id. Input is captured at chat-model start, before the provider call. A failed run never captures output.ChatMessage.blocks(core 0.12.3+) onLLMChatStartEventandLLMChatEndEvent:ImageBlock.image,AudioBlock.audio,DocumentBlock.data.IOBasesources are never read andpathis never opened.Message.images/audio/filescontent bytes at the model boundary (invoke,ainvoke, and the input of the stream variants), plus generatedModelResponsemedia for non-streaming calls. Privacy is the intersection of the FITraceConfigand the OpenInferenceTraceConfig: if either one hides, capture is denied.configure_agno_tracing(..., attachments=manager).BinaryContentin user prompts, in tool returns (on the tool span, withtool_call_id) and in re-sentFileParthistory. Also theFilePartoutput of a successful, non-streaming response. Error paths never capture.All four accept
attachments=(and auto'sattachment_manager=). With a missing, disabled or pre-attachments manager, every code path runs as before.One model call, one owner
fi_instrumentation.attachments.media_captureadds the framework helpers:attach_bytes,capture_parts+MediaPart,intersect(a deny-winsCapturePolicyintersection),sniff_mediaandlooks_base64. They are still not in__all__.Owner mark (new in
16311a37, binding rule of 2026-11-12). While attachments are enabled, each adapter runs the downstream provider SDK call undermedia_capture.OWNER_KEY.traceai_openaiandtraceai_anthropicthen upload nothing for that call. Framework +traceai_openaion one manager went from 2 uploads per part to 1, held by the framework span. This is proven withtraceai_openai(every adapter) and withtraceai_anthropicunder LangChainChatAnthropic(sync and stream: 1 upload, on the framework span; framework attachments off: 1, on the Anthropic span). Anthropic under the other adapters is not driven: Pydantic AI 2.67.0 callsclient.beta.messages.create(pydantic_ai/models/anthropic/__init__.py:1695), whichtraceai_anthropicdoes not wrap, so there is no second layer;llama-index-llms-anthropic0.12.3 fails withanthropic2.8.0 before sending any request (TypeError: Cannot instantiate typing.Union, also when traceAI is not imported). Every row sends one image part. Two parts are two positions, so they are two framework uploads; that case is not exercised.BaseChatModel.generate/agenerate/stream/astream(invoke, batch and transform go through these)chat/achat/stream_chat/astream_chatof every LLM class, including providers imported later. Completion calls are not observed, so they are not marked.invoke/ainvoke, and each advance ofinvoke_stream/ainvoke_stream(throughspan_scope's values)Model.request. Forrequest_stream, only entering and exiting the stream, never the caller'sasync withbodyowned_stream). The caller's code between chunks, or a stream it abandoned, is never marked.2296837f, review P1).16311a37marked every model call while attachments were enabled. An AgnoImage/File(filepath=...)or a LlamaIndexImageBlock/DocumentBlock(path=...)is skipped by the adapter by design, but the SDK integration inlines it as a data URI, andtraceai_openaiskipped the marked call: framework +traceai_openaion one manager (the auto default) went from 1 upload (provider span) to 0. Now each adapter marks a call only when its own parts scan yields at least one input media part (observes_agno_input,observes_langchain_input,observes_llamaindex_messages,PydanticAIMedia.observes_request). Those calls are back to 1 upload, on the provider span. Every gate fails open: a scan error means no mark, never a suppressed provider.hide_inputs,pii_redaction) is observed but not admitted. Gating on admission would leave that call unmarked and let the provider upload what the framework hid. Parts observed but rejected on size, magic bytes orRemoteUrlpolicy reach the same core checks in the provider.filepath/pathpart is marked, so the provider skips the path part too: 1 upload (the content part, on the framework span), not 2.filepath/pathsource is captured by the provider under the provider's ownTraceConfig. That is the base (f5e77e08) behaviour.16311a37.call_owner), and only for a call with an observed input part. With attachments off nothing is marked or patched, and tracing is the base path. A call that is already owned (an outer adapter, ortraceai.with_attachment_ownership) is not re-marked, and the adapter captures nothing for it.CallOwnerwith their namespace.capture_partstreats a mark of its own namespace as its own. Agno and Pydantic AI capture before marking, so their mark has no namespace: a nested request of the same framework (a wrapper or fallback model) is owned too.TraceConfighides inputs, the framework span uploads nothing, and the provider span beneath it uploads nothing either (see Residual 2 for path-only calls).provider_media.claim(the LiteLLM helper in feat(python): provider media for Gemini, Vertex, Bedrock, LiteLLM and compatible chat #697). That module is not on this stack. The mark is the same key with the same meaning, set throughmedia_capture.call_owner/owner_scope.2296837fchanges only a docstring inmedia_capture.py; the gate lives in the adapters.3fe575c4cherry-pickse1586827from feat(python): provider media for Gemini, Vertex, Bedrock, LiteLLM and compatible chat #697 (CAPTURE_POLICY_KEY,OWNER_KEY,owned_elsewhere,merge_capture_policies). Every non-docstring hunk is byte-identical to that commit and to feat(python): one shared attachment manager in the auto module (traceai) #692, so those merges stay clean here.Different calls still each own their occurrence:
fi-attachment:URI is only ever projected onto the span whose ownfi.attachment.refslists that occurrence;Known limit: a chat model that calls another chat model of the same framework inside its own call (LangChain or LlamaIndex) is still observed by both framework spans. The provider span still skips.
Changed in review (
9a3c7ca1,d1d18014,80693890on top of3ad81031). The earlier per-manager claim table (attach_claimed,CLAIMS,CLAIM_TTL) is withdrawn. It reused one occurrence across spans and traces. That gave a later span a URI its ownfi.attachment.refsdid not list, and when the first upload was dropped, the reuse pointed at nothing. The owner mark is a different mechanism: nothing is reused. The provider layer of the SAME call simply does not observe it.Fail closed
image_urlhttp(s), LlamaIndexurl, Agnourl, Pydantic AIImageUrl/AudioUrl/DocumentUrl) go to the coreRemoteUrlpolicy, which denies them today. Nothing here resolves or fetches them: a loopback server and a link-local URL see zero requests.externalobjects and media references, videos and unknown objects are never read. No object is serialized to look for media.attach_bytestakes a callable and calls it only after a preflight at the part cap has passed every privacy, correlation and sampling step. The decoded container (magic bytes) must match the admitted MIME (PNG, JPEG, WebP, PDF, WAV, MP3, Ogg, FLAC, WebM). Limits are 1 MiB per part and 64 positions per span, as in@traceai/framework-attachments.TraceConfighide flags andpii_redactiondeny for their direction, and the core cannot relax them, not even withallow_unredacted_media_with_text_pii.capture_parts(..., policy_directions=("input", "output"))therefore folds both directions, as in the TypeScript MCP binding:hide_outputs,hide_output_textorhide_output_messagesdeny it on the tool span just like the input flags do. It is still uploaded asdirection="input",purpose="tool_result".Not in this PR
fi-attachment:projection into framework span attributes. LangChain already redacts data URIs there; the other frameworks keep their existing serialization.traceai.init(), because the kwargs plumbing is the sameattachment_manager=path that feat(python): OpenAI, Anthropic and Groq attachments (images, files, computer-use screenshots, speech) #690 tests for OpenAI. (Agno and Pydantic AI joined the auto registry in feat(python): one shared attachment manager in the auto module (traceai) #692.)traceai_pydantic_aideclarespydantic-ai, so on apydantic-ai-slim-only installinstrument()is a silent no-op unlessskip_dep_check=True. This predates the PR; the tests pass the flag.Stacking
Stacked on
feat/attachments-python-openai-anthropic(#690), because it extends the shared helper #690 introduces.Tests
New suite
python/tests/attachments_frameworks(it reuses the #690 loopback F01 harness) and workflowattachments-python-frameworks-tests.yml.Owner gate (
2296837f):test_provider_and_framework.py, framework + real OpenAI SDK +traceai_openaion one manager:test_a_source_only_the_sdk_reads_is_still_captured_by_the_provider[agno-image|agno-file|llamaindex-image|llamaindex-document]: 1 upload, on the provider span, and the URI resolves. On1080f58dthese were 0.test_the_framework_alone_never_reads_a_path_source: the same calls withouttraceai_openaigive 0 uploads, while the SDK still sends the bytes.test_residual_a_call_mixing_an_observed_part_and_a_path_part_is_marked[agno|llamaindex]: 1 upload, on the framework span (Residual 1).test_owner_gate.py: probe models record the mark beneath LangChain and LlamaIndex. It is unset for text, provider-file and path-only calls, and set for an observed image (generate, stream, positional and keyword messages). Agno and Pydantic AI gate units, plus fail-open cases.2296837f: 17/17 checks pass. Framework attachments: latest 137 passed, 0 skipped (was 121); floor 110 passed, 27 skipped (was 99/22). The 5 new floor skips are the LlamaIndex OpenAI rows, which are latest-only. The Agno path rows ran on both tiers (agno 2.7.2 and 3.1.12).workflow_dispatchon temporary branches, deleted afterwards):1080f58d's behaviour: 10 failed on latest and 8 on floor, including every path row;hide_inputs, residual and abandoned-stream rows).Owner mark (
16311a37,1080f58d):test_provider_and_framework.py, real framework + real OpenAI SDK +traceai_openaion one manager:80693890these were 2.async withbody is not marked;hide_inputsgives 0 uploads, provider included.test_owner.py(core): gating, mark scope on exception, own-namespace capture versus anonymous, other-namespace and user marks, sync/async stream marking, an abandoned stream, and a proxy that implements both protocols.Earlier review fixes (unchanged):
fi-attachment:URI resolution invariant;hide_outputs/hide_output_text/hide_output_messagesnegative controls on the Pydantic AI tool span.Anthropic under a framework and pinned owner checks (
a2a007f2,66c43710, tests and CI only):test_anthropic_under_framework.py: LangChainChatAnthropic+ real Anthropic SDK +traceai_anthropicon one manager. Sync and stream give 1 upload, on the framework span, and the URI resolves. With framework attachments off, the Anthropic span owns the 1 upload. The workflow's latest tier now installstraceai_anthropic,langchain-anthropic1.7.10 andanthropic2.8.0; the floor tier pinsanthropic0.41.0.test_owner.py::test_each_provider_side_check_honours_the_mark: the provider skips an owned call through two checks,Observation.usableandlocal_capture_policy(...).requested. Ona2a007f2, removing either check alone failed no test, because the other check still skipped the call; removing both failed 31. Each check now has its own assertion.test_owner_gate.py::test_pydantic_ai_call_owner_follows_the_gate: the Pydantic AI wrapper's_call_ownergives no mark for a text-only request. Before this test, removing that gate failed no test.66c43710: 17/17 checks pass. Framework attachments: latest 142 passed; floor 112 passed, 30 skipped.VM proof on
66c43710(private GCP VM, outbound network removed withunshare --net, TLS loopback F01):langchain-core0.2.43,llama-index-core0.12.41,agno2.7.2,pydantic-ai-slim0.1.0,openai1.74.0,anthropic0.41.0, OTel 1.38) 112 passed, 30 skipped. Latest (py3.12,langchain-core1.8.0 +langchain-openai1.7.4 +langchain-anthropic1.7.10,llama-index-core0.14.25 +llama-index-llms-openai0.8.4,agno3.1.12,pydantic-ai-slim2.67.0,openai3.50.0,anthropic2.8.0) 142 passed.80693890(before the owner mark): floor 25 failed, latest 44 failed. The failures are the one-owner rows (2 uploads, including both Anthropic rows), the two-call,hide_inputs, abandoned-stream, stream-body and residual rows, plus the core helper and gate units the old head does not have.1080f58d(mark on every call): floor 9 failed, latest 11 failed. These are only the owner-gate units and the path-source rows (0 uploads there).f5e77e08on the same VM: identical failure sets on both tiers for provider attachments (57 passed, 1 skipped), core attachments (410), and the langchain, llama_index, agno, pydantic-ai, openai and anthropic packages. Counts: floor langchain 108+8F, llama_index 87, agno 706+40F, pydantic-ai 268+40F, openai 1821, anthropic 162. Latest langchain 119, llama_index 138+12F, agno 746, pydantic-ai 268+40F, openai 1845, anthropic 246. The failures predate this PR.capture_partsis suppressed by its own mark.owned_streamreturns the bare generator, or the mark is left attached across a yield.call_ownerignores attachments being off.Earlier CI (before the VM proof above):
1080f58d: 17/17 checks pass.langchain-openai1.7.4,llama-index-llms-openai0.8.4,agno3.1.12 andpydantic-ai-slim2.67.0 against the real OpenAI SDK.16311a37failed exactly one row:llamaindex-astream. LlamaIndex's async stream proxy also implements the sync protocol, and the stream was marked as a sync generator.1080f58dchecks the async protocol first and adds a core test for it.Earlier VM evidence (head
80693890, before the owner mark): Every run below was on a private GCP VM with outbound network removed (unshare --net, loopback only), using real framework SDKs and a TLS loopback F01 server. Three trees ran side by side: this head, the pre-review head3ad81031and the basef5e77e08.3ad81031langchain-core0.2.43,llama-index-core0.12.41,agno2.7.2,pydantic-ai-slim0.1.0,openai1.74.0, OTel < 1.39langchain-core1.8.0 +langchain-openai1.7.4,llama-index-core0.14.25 +llama-index-llms-openai0.8.4,agno3.1.12,pydantic-ai2.67.0,openai3.49.03ad81031:traceai_openainow leave no dangling URI. Before: 1.hide_outputs/hide_output_text/hide_output_messages: no tool-span upload now, where before the tool span uploaded the screenshot. Only the later model request (input-governed) keeps a copy.main/dev, so these VM runs are the package regression evidence.test_occurrences.pyran 15 times per tier on the final head and passed 21/21 every time. One earlier run showed that the two parents' upload order is not fixed (concurrent workers);80693890stopped asserting it.policy_directionsignored;tool_call_id;Earlier CI on
80693890(and ond1d18014before it): 17/17 checks pass. Framework attachments: floor 81 passed, 4 skipped; latest 85 passed. Provider attachments: floor 63 passed, 2 skipped; latest 65 passed. On9a3c7ca1CI caught 3 test-side errors and one real gap: an in-memory bytes part was read before its preflight.d1d18014fixes all four.🤖 Generated with Claude Code