Skip to content

feat(python): framework media capture, one owner per model call (LangChain, LlamaIndex, Agno, Pydantic AI) - #691

Draft
nik13 wants to merge 17 commits into
feat/attachments-python-openai-anthropicfrom
feat/attachments-python-frameworks
Draft

nik13 wants to merge 17 commits into
feat/attachments-python-openai-anthropicfrom
feat/attachments-python-frameworks

Conversation

@nik13

@nik13 nik13 commented Nov 11, 2026 •

Copy link
Copy Markdown
Contributor

Part of TH-2379 (Linear).

What this does

Framework instrumentors now capture typed media through the shared attachment manager. One model call, one owner: when a framework adapter observes a model call's input media, the provider instrumentor beneath it skips that call, so each part is uploaded once. A call whose media the adapter never reads (an Agno filepath, a LlamaIndex path) is not marked, so the provider span still captures what the SDK inlines. Different calls each get their own occurrence.

  • LangChain (F02): chat-model input and output message content (the dumpd form the tracer receives): OpenAI-style image_url / file / input_audio data, LangChain 0.3 source_type: "base64" blocks, LangChain 1.x base64 content blocks and Anthropic-style source blocks. ToolMessage media keeps its tool_call_id. Input is captured at chat-model start, before the provider call. A failed run never captures output.
  • LlamaIndex (F02): ChatMessage.blocks (core 0.12.3+) on LLMChatStartEvent and LLMChatEndEvent: ImageBlock.image, AudioBlock.audio, DocumentBlock.data. IOBase sources are never read and path is never opened.
  • Agno (F08): Message.images / audio / files content bytes at the model boundary (invoke, ainvoke, and the input of the stream variants), plus generated ModelResponse media for non-streaming calls. Privacy is the intersection of the FI TraceConfig and the OpenInference TraceConfig: if either one hides, capture is denied. configure_agno_tracing(..., attachments=manager).
  • Pydantic AI (F09): BinaryContent in user prompts, in tool returns (on the tool span, with tool_call_id) and in re-sent FilePart history. Also the FilePart output of a successful, non-streaming response. Error paths never capture.

All four accept attachments= (and auto's attachment_manager=). With a missing, disabled or pre-attachments manager, every code path runs as before.

One model call, one owner

fi_instrumentation.attachments.media_capture adds the framework helpers: attach_bytes, capture_parts + MediaPart, intersect (a deny-wins CapturePolicy intersection), sniff_media and looks_base64. They are still not in __all__.

Owner mark (new in 16311a37, binding rule of 2026-11-12). While attachments are enabled, each adapter runs the downstream provider SDK call under media_capture.OWNER_KEY. traceai_openai and traceai_anthropic then upload nothing for that call. Framework + traceai_openai on one manager went from 2 uploads per part to 1, held by the framework span. This is proven with traceai_openai (every adapter) and with traceai_anthropic under LangChain ChatAnthropic (sync and stream: 1 upload, on the framework span; framework attachments off: 1, on the Anthropic span). Anthropic under the other adapters is not driven: Pydantic AI 2.67.0 calls client.beta.messages.create (pydantic_ai/models/anthropic/__init__.py:1695), which traceai_anthropic does not wrap, so there is no second layer; llama-index-llms-anthropic 0.12.3 fails with anthropic 2.8.0 before sending any request (TypeError: Cannot instantiate typing.Union, also when traceAI is not imported). Every row sends one image part. Two parts are two positions, so they are two framework uploads; that case is not exercised.

Adapter What runs under the mark
LangChain BaseChatModel.generate / agenerate / stream / astream (invoke, batch and transform go through these)
LlamaIndex chat / achat / stream_chat / astream_chat of every LLM class, including providers imported later. Completion calls are not observed, so they are not marked.
Agno the wrapped invoke / ainvoke, and each advance of invoke_stream / ainvoke_stream (through span_scope's values)
Pydantic AI Model.request. For request_stream, only entering and exiting the stream, never the caller's async with body
  • Never across a yield. A returned stream is marked only while it advances (owned_stream). The caller's code between chunks, or a stream it abandoned, is never marked.
  • Mark only what the adapter observes (2296837f, review P1). 16311a37 marked every model call while attachments were enabled. An Agno Image/File(filepath=...) or a LlamaIndex ImageBlock/DocumentBlock(path=...) is skipped by the adapter by design, but the SDK integration inlines it as a data URI, and traceai_openai skipped the marked call: framework + traceai_openai on one manager (the auto default) went from 1 upload (provider span) to 0. Now each adapter marks a call only when its own parts scan yields at least one input media part (observes_agno_input, observes_langchain_input, observes_llamaindex_messages, PydanticAIMedia.observes_request). Those calls are back to 1 upload, on the provider span. Every gate fails open: a scan error means no mark, never a suppressed provider.
    • Deviation from the review's wording, disclosed: the gate is on parts the adapter OBSERVES, not on parts it ADMITS. A part the framework hides (hide_inputs, pii_redaction) is observed but not admitted. Gating on admission would leave that call unmarked and let the provider upload what the framework hid. Parts observed but rejected on size, magic bytes or RemoteUrl policy reach the same core checks in the provider.
    • Residual 1 (pinned by a test): a call that mixes an observed part with a filepath/path part is marked, so the provider skips the path part too: 1 upload (the content part, on the framework span), not 2.
    • Residual 2: a framework hide flag reaches only calls the adapter marks. A call whose only media is a filepath/path source is captured by the provider under the provider's own TraceConfig. That is the base (f5e77e08) behaviour.
    • Residual 3: a call with no observed input part but with generated output media is unmarked, so the framework span and the provider span may both upload that output. That was the behaviour before 16311a37.
  • Gating. The mark is set only while the adapter's manager can admit (call_owner), and only for a call with an observed input part. With attachments off nothing is marked or patched, and tracing is the base path. A call that is already owned (an outer adapter, or traceai.with_attachment_ownership) is not re-marked, and the adapter captures nothing for it.
  • The adapter still captures inside its own mark. LangChain and LlamaIndex capture in callbacks that fire inside the marked call, so their mark is a CallOwner with their namespace. capture_parts treats a mark of its own namespace as its own. Agno and Pydantic AI capture before marking, so their mark has no namespace: a nested request of the same framework (a wrapper or fallback model) is owned too.
  • Deny carries down for a marked call. If the framework's TraceConfig hides inputs, the framework span uploads nothing, and the provider span beneath it uploads nothing either (see Residual 2 for path-only calls).
  • Deviation, disclosed: the task named provider_media.claim (the LiteLLM helper in feat(python): provider media for Gemini, Vertex, Bedrock, LiteLLM and compatible chat #697). That module is not on this stack. The mark is the same key with the same meaning, set through media_capture.call_owner / owner_scope.
  • 2296837f changes only a docstring in media_capture.py; the gate lives in the adapters.
  • 3fe575c4 cherry-picks e1586827 from feat(python): provider media for Gemini, Vertex, Bedrock, LiteLLM and compatible chat #697 (CAPTURE_POLICY_KEY, OWNER_KEY, owned_elsewhere, merge_capture_policies). Every non-docstring hunk is byte-identical to that commit and to feat(python): one shared attachment manager in the auto module (traceai) #692, so those merges stay clean here.

Different calls still each own their occurrence:

  • the same bytes in two framework calls are 2 uploads, on 2 framework spans;
  • a framework call followed by a direct SDK call is 2 uploads, one on each span;
  • byte deduplication is left to the server;
  • a fi-attachment: URI is only ever projected onto the span whose own fi.attachment.refs lists that occurrence;
  • the per-span registry stays: observing the same span and position twice gives one occurrence.

Known limit: a chat model that calls another chat model of the same framework inside its own call (LangChain or LlamaIndex) is still observed by both framework spans. The provider span still skips.

Changed in review (9a3c7ca1, d1d18014, 80693890 on top of 3ad81031). The earlier per-manager claim table (attach_claimed, CLAIMS, CLAIM_TTL) is withdrawn. It reused one occurrence across spans and traces. That gave a later span a URI its own fi.attachment.refs did not list, and when the first upload was dropped, the reuse pointed at nothing. The owner mark is a different mechanism: nothing is reused. The provider layer of the SAME call simply does not observe it.

Fail closed

  • Remote URLs (LangChain image_url http(s), LlamaIndex url, Agno url, Pydantic AI ImageUrl / AudioUrl / DocumentUrl) go to the core RemoteUrl policy, which denies them today. Nothing here resolves or fetches them: a loopback server and a link-local URL see zero requests.
  • Paths, open streams, provider file IDs, Agno external objects and media references, videos and unknown objects are never read. No object is serialized to look for media.
  • Every part is preflighted before it is read or decoded. This now includes in-memory bytes parts: attach_bytes takes a callable and calls it only after a preflight at the part cap has passed every privacy, correlation and sampling step. The decoded container (magic bytes) must match the admitted MIME (PNG, JPEG, WebP, PDF, WAV, MP3, Ogg, FLAC, WebM). Limits are 1 MiB per part and 64 positions per span, as in @traceai/framework-attachments.
  • Local TraceConfig hide flags and pii_redaction deny for their direction, and the core cannot relax them, not even with allow_unredacted_media_with_text_pii.
  • Pydantic AI tool results: media returned by a tool is the tool span's output (that span redacts its result under the output flags) and the next request's model input. capture_parts(..., policy_directions=("input", "output")) therefore folds both directions, as in the TypeScript MCP binding: hide_outputs, hide_output_text or hide_output_messages deny it on the tool span just like the input flags do. It is still uploaded as direction="input", purpose="tool_result".

Not in this PR

Stacking

Stacked on feat/attachments-python-openai-anthropic (#690), because it extends the shared helper #690 introduces.

Tests

New suite python/tests/attachments_frameworks (it reuses the #690 loopback F01 harness) and workflow attachments-python-frameworks-tests.yml.

Owner gate (2296837f):

  • test_provider_and_framework.py, framework + real OpenAI SDK + traceai_openai on one manager:
    • test_a_source_only_the_sdk_reads_is_still_captured_by_the_provider[agno-image|agno-file|llamaindex-image|llamaindex-document]: 1 upload, on the provider span, and the URI resolves. On 1080f58d these were 0.
    • negative control test_the_framework_alone_never_reads_a_path_source: the same calls without traceai_openai give 0 uploads, while the SDK still sends the bytes.
    • test_residual_a_call_mixing_an_observed_part_and_a_path_part_is_marked[agno|llamaindex]: 1 upload, on the framework span (Residual 1).
  • test_owner_gate.py: probe models record the mark beneath LangChain and LlamaIndex. It is unset for text, provider-file and path-only calls, and set for an observed image (generate, stream, positional and keyword messages). Agno and Pydantic AI gate units, plus fail-open cases.
  • CI on 2296837f: 17/17 checks pass. Framework attachments: latest 137 passed, 0 skipped (was 121); floor 110 passed, 27 skipped (was 99/22). The 5 new floor skips are the LlamaIndex OpenAI rows, which are latest-only. The Agno path rows ran on both tiers (agno 2.7.2 and 3.1.12).
  • Mutations (CI workflow_dispatch on temporary branches, deleted afterwards):
    • always mark, which is 1080f58d's behaviour: 10 failed on latest and 8 on floor, including every path row;
    • never mark: 32 failed on latest and 15 on floor (one-owner, hide_inputs, residual and abandoned-stream rows).

Owner mark (16311a37, 1080f58d):

  • test_provider_and_framework.py, real framework + real OpenAI SDK + traceai_openai on one manager:
    • one call = exactly 1 upload, under the framework span, and the provider span holds none. Covered for LangChain sync/stream/async/astream, LlamaIndex sync/stream/async/astream, Agno sync/stream and Pydantic AI sync/stream. On 80693890 these were 2.
    • framework alone still uploads 1 (control: the adapter is not suppressed by its own mark);
    • attachments off for the framework: nothing is marked, and the provider uploads 1;
    • the same bytes in two framework calls: 2;
    • a framework call, then a direct SDK call: 2;
    • an abandoned LangChain or LlamaIndex stream leaves no mark: a direct call while it is suspended, and another after it is closed, each upload;
    • the Pydantic AI caller's async with body is not marked;
    • framework hide_inputs gives 0 uploads, provider included.
  • test_owner.py (core): gating, mark scope on exception, own-namespace capture versus anonymous, other-namespace and user marks, sync/async stream marking, an abandoned stream, and a proxy that implements both protocols.

Earlier review fixes (unchanged):

  • two unmarked spans give two occurrences, each referenced on its own span (raw + inline, two traces, concurrent observers, failed upload never reused);
  • a provider-only two-call regression;
  • a fi-attachment: URI resolution invariant;
  • denied parts are never read;
  • hide_outputs / hide_output_text / hide_output_messages negative controls on the Pydantic AI tool span.

Anthropic under a framework and pinned owner checks (a2a007f2, 66c43710, tests and CI only):

  • test_anthropic_under_framework.py: LangChain ChatAnthropic + real Anthropic SDK + traceai_anthropic on one manager. Sync and stream give 1 upload, on the framework span, and the URI resolves. With framework attachments off, the Anthropic span owns the 1 upload. The workflow's latest tier now installs traceai_anthropic, langchain-anthropic 1.7.10 and anthropic 2.8.0; the floor tier pins anthropic 0.41.0.
  • test_owner.py::test_each_provider_side_check_honours_the_mark: the provider skips an owned call through two checks, Observation.usable and local_capture_policy(...).requested. On a2a007f2, removing either check alone failed no test, because the other check still skipped the call; removing both failed 31. Each check now has its own assertion.
  • test_owner_gate.py::test_pydantic_ai_call_owner_follows_the_gate: the Pydantic AI wrapper's _call_owner gives no mark for a text-only request. Before this test, removing that gate failed no test.
  • CI on 66c43710: 17/17 checks pass. Framework attachments: latest 142 passed; floor 112 passed, 30 skipped.

VM proof on 66c43710 (private GCP VM, outbound network removed with unshare --net, TLS loopback F01):

  • Framework suite on this head: floor (py3.10, langchain-core 0.2.43, llama-index-core 0.12.41, agno 2.7.2, pydantic-ai-slim 0.1.0, openai 1.74.0, anthropic 0.41.0, OTel 1.38) 112 passed, 30 skipped. Latest (py3.12, langchain-core 1.8.0 + langchain-openai 1.7.4 + langchain-anthropic 1.7.10, llama-index-core 0.14.25 + llama-index-llms-openai 0.8.4, agno 3.1.12, pydantic-ai-slim 2.67.0, openai 3.50.0, anthropic 2.8.0) 142 passed.
  • This head's tests on older heads:
    • On 80693890 (before the owner mark): floor 25 failed, latest 44 failed. The failures are the one-owner rows (2 uploads, including both Anthropic rows), the two-call, hide_inputs, abandoned-stream, stream-body and residual rows, plus the core helper and gate units the old head does not have.
    • On 1080f58d (mark on every call): floor 9 failed, latest 11 failed. These are only the owner-gate units and the path-source rows (0 uploads there).
  • Package suites, this head vs base f5e77e08 on the same VM: identical failure sets on both tiers for provider attachments (57 passed, 1 skipped), core attachments (410), and the langchain, llama_index, agno, pydantic-ai, openai and anthropic packages. Counts: floor langchain 108+8F, llama_index 87, agno 706+40F, pydantic-ai 268+40F, openai 1821, anthropic 162. Latest langchain 119, llama_index 138+12F, agno 746, pydantic-ai 268+40F, openai 1845, anthropic 246. The failures predate this PR.
  • Owner mutations: 16/16 caught against the 142-row latest suite, with a passing control:
    • O1, O1b, O1c: the provider ignores the mark, through either check or both. Both together fail 32 rows, including the Anthropic rows.
    • O2: capture_parts is suppressed by its own mark.
    • O3, O4: owned_stream returns the bare generator, or the mark is left attached across a yield.
    • O5: call_owner ignores attachments being off.
    • O6-O9: an adapter does not mark (LangChain, LlamaIndex, Agno, Pydantic AI).
    • O10: Pydantic AI marks the caller's stream body.
    • O11-O14: an adapter marks a call it does not observe (each adapter's gate removed).

Earlier CI (before the VM proof above):

  • CI on 1080f58d: 17/17 checks pass.
    • Framework attachments, latest tier: 121 passed, 0 skipped. Every new row ran, with real langchain-openai 1.7.4, llama-index-llms-openai 0.8.4, agno 3.1.12 and pydantic-ai-slim 2.67.0 against the real OpenAI SDK.
    • Framework attachments, floor tier: 99 passed, 22 skipped. The skips are the LangChain and LlamaIndex OpenAI rows, because those integrations are installed on latest only. The Agno and Pydantic AI rows ran on the floor tier.
  • 16311a37 failed exactly one row: llamaindex-astream. LlamaIndex's async stream proxy also implements the sync protocol, and the stream was marked as a sync generator. 1080f58d checks the async protocol first and adds a core test for it.

Earlier VM evidence (head 80693890, before the owner mark): Every run below was on a private GCP VM with outbound network removed (unshare --net, loopback only), using real framework SDKs and a TLS loopback F01 server. Three trees ran side by side: this head, the pre-review head 3ad81031 and the base f5e77e08.

Tier Versions New suite (head) New tests on 3ad81031
floor (py3.10) langchain-core 0.2.43, llama-index-core 0.12.41, agno 2.7.2, pydantic-ai-slim 0.1.0, openai 1.74.0, OTel < 1.39 81 passed, 4 skipped 14 failed
latest (py3.12) langchain-core 1.8.0 + langchain-openai 1.7.4, llama-index-core 0.14.25 + llama-index-llms-openai 0.8.4, agno 3.1.12, pydantic-ai 2.67.0, openai 3.49.0 85 passed 16 failed
  • Review probes, head vs 3ad81031:
    • Two independent traces with the same image now give 2 uploads, each span's URI in its own refs. Before: 1 upload, and the second span's URI was not in its refs.
    • LangChain + traceai_openai now leave no dangling URI. Before: 1.
    • A dropped first upload is no longer reused: 1 payload after restore, before 0.
    • Pydantic AI tool span under hide_outputs / hide_output_text / hide_output_messages: no tool-span upload now, where before the tool span uploaded the screenshot. Only the later model request (input-governed) keeps a copy.
  • Regressions, head vs base on the same VM: identical counts and identical failure sets in every package suite. Floor: langchain 108+8F, llama_index 87, agno 706+40F, pydantic-ai 268+40F. Latest: langchain 119, llama_index 138+12F, agno 746, pydantic-ai 308. Provider attachments 57+1s and core attachments 410 on both tiers. The per-package workflows only run for PRs into main/dev, so these VM runs are the package regression evidence.
  • Stability: test_occurrences.py ran 15 times per tier on the final head and passed 21/21 every time. One earlier run showed that the two parents' upload order is not fixed (concurrent workers); 80693890 stopped asserting it.
  • Mutations: the control passed and 12/12 mutations were each caught by a named test:
    • cross-span occurrence reuse reintroduced;
    • policy_directions ignored;
    • tool result folding input flags only;
    • a bytes part read before preflight;
    • local TraceConfig fold;
    • magic-byte check;
    • part limit;
    • LangChain TraceConfig;
    • Agno config intersection;
    • Pydantic AI tool_call_id;
    • LlamaIndex output capture;
    • LangChain input capture.
  • Residual, not changed here: the per-span registry lock is not reset at fork, so a child forked while another thread holds it would block on its first capture. This is unchanged from base, and the withdrawn claim lock had the same property.

Earlier CI on 80693890 (and on d1d18014 before it): 17/17 checks pass. Framework attachments: floor 81 passed, 4 skipped; latest 85 passed. Provider attachments: floor 63 passed, 2 skipped; latest 65 passed. On 9a3c7ca1 CI caught 3 test-side errors and one real gap: an in-memory bytes part was read before its preflight. d1d18014 fixes all four.

🤖 Generated with Claude Code

nik13 and others added 11 commits November 11, 2026 09:33
LangChain and LlamaIndex message content blocks, Agno Image/Audio/File and
Pydantic AI BinaryContent reach the shared attachment manager. The shared
helper gains a per-trace claim table, so a framework span and the provider
span beneath it (traceai_openai, traceai_anthropic) upload identical bytes
once. A later observer must pass its own policy before reusing a claim, and
only admitted bytes are hashed.

URLs go to the core RemoteUrl policy and are never fetched; paths, streams,
provider files and unknown objects are never read. Agno capture uses the
intersection of the FI and OpenInference TraceConfig.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…c-ai 0.1.0 needs

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
LangChain and LlamaIndex do not make their spans current, so the provider
span beneath them starts a separate trace. Claims are now keyed by manager,
MIME and digest with a lazy 300 s expiry, so the framework and provider
observations of one call still upload once. Tests cover separate traces,
another manager and claim expiry; pydantic-ai tests skip the dependency
check because CI installs pydantic-ai-slim.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Patching time.monotonic globally also moved the manager's own deadlines.
The floor tier pins an OpenTelemetry SDK that pydantic-ai 0.1.0 can import.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… and installs fastapi

pydantic-ai 0.1.0 imports opentelemetry._events, which API 1.39+ no longer
ships, and Agno imports fastapi from its workflow module.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ent results

pydantic-ai 0.1.0 hangs in a forked child with no instrumentation at all, so
the fork exit probe runs on LangChain only. Its FunctionModel also cannot
serialize a binary tool return on the second request; the tool span has
captured the image by then, which the test still asserts.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…i 0.x

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…h hide directions

Remove the per-manager claim table from media_capture. It reused one
occurrence across spans and traces, so a later span carried a
fi-attachment URI that its own fi.attachment.refs did not list and that
F01 resolve (which checks the expected parent) could never resolve.
Every observing span now uploads and references its own occurrence;
byte deduplication is left to the server. This also removes the
dropped-upload reuse and the claim lock held across fork.

capture_parts gains policy_directions. The Pydantic AI tool span folds
both the input and the output hide flags (deny-wins) for the media a
tool returns, because that span redacts its result under the output
flags and the result is also a later model input (as in the TS MCP
binding).

Tests: claim tests inverted (two spans, two occurrences, each
referenced on its own span); provider-only two-call regression; a
fi-attachment URI resolution invariant on every real-provider test;
hide_outputs / hide_output_text / hide_output_messages negative
controls on the tool span.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…orkflow comment

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…currence tests

capture_parts handed attach_bytes the bytes of a "bytes" part before any
preflight, so a part denied by a local hide flag was still read.
attach_bytes now also takes a zero-argument callable and calls it only
after a preflight at the part cap passed every privacy, correlation and
sampling step (too_large / queue_full still size exactly, as in
attach_inline).

Tests: span refs carry purpose and indices, not tool_call_id (that is in
the upload request), and a provider span projects its fi-attachment URI
into several attributes, so the provider-only test now checks that every
URI resolves to its own span and each span carries its own.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Upload workers run concurrently, so the two parents can reach the
loopback F01 server in either order (seen once in 15 runs on a VM).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@nik13 nik13 changed the title feat(python): framework media capture with one upload per trace (LangChain, LlamaIndex, Agno, Pydantic AI) feat(python): framework media capture, one occurrence per observing span (LangChain, LlamaIndex, Agno, Pydantic AI) Nov 12, 2026
nik13 and others added 3 commits November 12, 2026 07:07
Cherry-picked from e158682 (feat/attachments-python-providers, #697). Every
non-docstring hunk of media_capture.py is byte-identical to that commit (and to
feat/attachments-python-auto), so merging those branches later is a no-op here.
The docstring keeps both bullets; the next commit rewrites the per-span one.

(cherry picked from commit e158682)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
When the LangChain, LlamaIndex, Agno or Pydantic AI adapter captures media for a
model call (attachments enabled), the downstream provider SDK call now runs under
the owner mark (media_capture.OWNER_KEY), so traceai_openai / traceai_anthropic
beneath it upload nothing for that call. Framework + OpenAI on one manager goes
from 2 uploads per part to 1, held by the framework span.

- core: CallOwner, call_owner (gated on an active manager and an unowned call),
  owner_scope / owned_call, and owned_stream, which marks each advance of a
  returned (async) generator and never the caller's code between chunks.
  capture_parts treats a CallOwner of its own namespace as its own, so the
  LangChain and LlamaIndex callbacks still capture inside the mark.
- LangChain: BaseChatModel generate/agenerate/stream/astream marked (patched only
  while attachments are enabled; restored on uninstrument).
- LlamaIndex: chat/achat/stream_chat/astream_chat of every LLM class (patched like
  StreamInstrumentation, including later-imported providers).
- Agno: the owner mark joins span_scope's values (call and each stream advance).
- Pydantic AI: request marked; request_stream only while entered and exited, never
  the caller's async-with body.
- DIFFERENT calls still each own their occurrence (same bytes twice = 2).
- Attachments off: nothing marked or patched; tracing is the base path.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…arking it

LlamaIndex's async stream proxy also defines send/throw/close, so owned_stream
checked it as a sync generator first (CI: llamaindex-astream TypeError).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@nik13 nik13 changed the title feat(python): framework media capture, one occurrence per observing span (LangChain, LlamaIndex, Agno, Pydantic AI) feat(python): framework media capture, one owner per model call (LangChain, LlamaIndex, Agno, Pydantic AI) Nov 12, 2026
nik13 and others added 3 commits November 12, 2026 08:09
…es its media

16311a3 set the owner mark on every framework model call while attachments were
enabled. Media the adapter deliberately skips was then captured by nobody: an Agno
Image/File(filepath=...) or a LlamaIndex ImageBlock/DocumentBlock(path=...) is
inlined as a data URI by the SDK integration, but traceai_openai skipped the marked
call. Framework + traceai_openai on one manager (the auto default) went from 1
upload (provider span) to 0.

- Agno, LangChain, LlamaIndex and Pydantic AI now mark a call only when the
  adapter's own parts scan yields at least one input media part
  (observes_agno_input / observes_langchain_input / observes_llamaindex_messages /
  PydanticAIMedia.observes_request). Every gate fails open: an error means no mark.
- The gate is on OBSERVED parts, not admitted ones: a part the framework hides
  (hide_inputs, pii_redaction) is still observed, so the mark keeps carrying the
  framework's deny to the provider (test_framework_hide_inputs_... stays 0).
- Regression rows (framework + traceai_openai): Agno filepath image/file and
  LlamaIndex path image/document -> 1 upload, on the provider span; framework
  alone -> 0 (negative control); a call mixing a content part and a path part ->
  1 upload on the framework span (disclosed residual: the path part is skipped).
- Probe rows: the provider call beneath LangChain/LlamaIndex is unmarked for text,
  provider-file and path-only calls, and marked for an observed image.
- media_capture.py: docstring only.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
LangChain ChatAnthropic + traceai_anthropic on one manager (sync and stream):
one upload, on the framework span; with framework attachments off the
Anthropic span owns it. Latest tier only; CI installs traceai_anthropic,
langchain-anthropic 1.7.10 and anthropic 2.8.0 (floor pins anthropic 0.41.0).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…gate

VM mutation run on a2a007f left two survivors:
- dropping the owner check from Observation alone (local_capture_policy
  still sets requested=False, so no row changed); each check now has its
  own assertion;
- Pydantic AI _call_owner marking a request it does not observe; now
  asserted directly (text-only request -> no mark).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant