Skip to content

feat(python): one shared attachment manager in the auto module (traceai) - #692

Draft
nik13 wants to merge 19 commits into
feat/attachments-python-openai-anthropicfrom
feat/attachments-python-auto
Draft

nik13 wants to merge 19 commits into
feat/attachments-python-openai-anthropicfrom
feat/attachments-python-auto

Conversation

@nik13

@nik13 nik13 commented Nov 11, 2026 •

Copy link
Copy Markdown
Contributor

Part of TH-2379 (Linear).

What this does

traceai.init (the Python auto module) now hands one attachment manager to every instrumentor it enables, as attachments=. This mirrors @traceai/auto (#687) where Python allows it. Python has no HTTP wire layer, so the TS wire capture (A03/A04) has no Python counterpart.

  • One shared manager. It is the provider's manager, or one the application lends with init(attachment_manager=...). The instrumentors also receive it as attachment_manager=, the alias feat(python): OpenAI, Anthropic and Groq attachments (images, files, computer-use screenshots, speech) #690 accepts.
  • get_attachment_manager() returns that manager, or None before init().
  • A borrowed manager can only gain restrictions. Auto adds the trace_config hide and PII flags to it and never removes a flag that is already set. Auto never shuts a borrowed manager down. When borrowing, the provider is built with attachments disabled, so FI_ATTACHMENTS_ENABLED=true cannot start a second manager. attachment_manager and attachment_config cannot be passed together.
  • Kill switch. With FI_ATTACHMENTS_ENABLED=false at init, instrumentors get the OFF arguments. If it is set after init, the core already denies every admission, and report() / attachment_status() now show disabled; a running manager's own status() does not consult the variable. A borrowed manager is not disabled permanently.
  • Per-call helpers:
    • with_attachment_capture_policy(policy, context=None) adds a CapturePolicy restriction. It is intersected deny-wins with any policy already in the context; anything that is not a CapturePolicy denies.
    • with_attachment_ownership(context=None) marks calls whose media your own adapter captures, so instrumentors upload nothing for them.
    • Both return an OpenTelemetry context. The keys are plain strings (traceai.auto.attachment_policy.v1, traceai.auto.attachment_owner.v1), the same strings as the TypeScript traceai_auto keys. Because the keys are plain strings, no package has to import another; code inside this repo should still use media_capture.CAPTURE_POLICY_KEY / OWNER_KEY rather than repeat the literals.
  • report() gives a content-free view:
    • attachments: state, queue counters, borrowed, sources, and what is unsupported.
    • integrations[]: one row per instrumentor package, with status and media. media is provider_adapter only when the installed instrumentor exposes its capability descriptor: for example traceai_openai, traceai_anthropic (rows also list media_operations, for example tool_result.input) and traceai_groq; the full list is in the sections below. It is session_binding for Pipecat/LiveKit when the installed package has the bind_* entry point, and unqualified otherwise.
    • It contains no endpoint, header or credential, and it is not a playback claim.

Cross-lane touch (shared helper from #690)

fi_instrumentation/attachments/media_capture.py gains a small, additive reader for the two context keys:

  • local_capture_policy folds the context policy in deny-only, and an owner mark sets requested=False.
  • Observation is unusable under an owner mark.
  • New public helpers there: merge_capture_policies, context_capture_policy, owned_elsewhere, CAPTURE_POLICY_KEY and OWNER_KEY.

All three Python provider adapters (OpenAI, Anthropic, Groq) go through this helper, so they honour both helpers with no adapter edits.

With attachments OFF, nothing changes

When the manager is the disabled null object (or killed by the environment variable), instrument() receives exactly the keyword arguments it received before: tracer_provider, config, attachment_manager, attachment_policy. The existing unit test that pins them is unchanged. VM proof: the same OFF probe on base and head exports identical spans and provider requests, starts no fi- thread, and registers the same fork hooks as base, none of them from attachments (see Testing).

MCP and agent media adapters (merged from #693)

This branch includes #693 (feat/attachments-python-mcp-agents) by merge, now at 4901f03d (per-client Claude tool-span deferral, the live-conversation gate and more image signatures; merged clean in 50370aa4). #693's MCP/agent workflow also runs on this PR.

  • Registry. auto_instrument accepts openai_agents / openai-agents, claude_agent_sdk / claude-agent-sdk, smolagents and firecrawl. Each one gets an explicit entry point (OpenAIAgentsInstrumentor, ClaudeAgentInstrumentor, SmolagentsInstrumentor, FirecrawlInstrumentor) and the same keyword arguments as every other instrumentor: the shared manager when capture is on, and exactly the OFF arguments when it is off. Nothing new is enabled unless it is named.
  • mcp_use / mcp-use. mcp-use has a LangChain callback, not an instrumentor, so enabling it only records it (like Pipecat and LiveKit). traceai.mcp_use_callback(**kwargs) returns a FutureAGICallback that uses auto's provider and trace_config hide/PII flags.
  • traceai.wrap_mcp_client(session, **kwargs). This is the MCP entry point that captures media; the MCP instrumentor only propagates context. An installed traceAI-mcp without wrap_mcp_client gets the session back unchanged, and the row reports unsupported_version.
  • Shared manager in the helpers. Both helpers pass attachments= only while capture is on and the installed package has the adapter. Otherwise they make the call they made before. Caller keyword arguments always win.
  • report(). _MEDIA_ADAPTERS has the six descriptors: mcp_attachment_capabilities, mcp_use_attachment_capabilities, openai_agents_attachment_capabilities, claude_agent_attachment_capabilities, smolagents_attachment_capabilities and firecrawl_attachment_capabilities. A row is provider_adapter with that package's media_operations only when the installed version has the descriptor; otherwise it is unqualified. This is TS parity: the name is in the adapter set. For mcp and mcp_use, the row describes the explicit entry point above, not auto on its own.

Provider and framework media adapters in auto (c53acf38)

This makes auto's registry and report() cover the adapters from #697 (provider media) and #691 (framework media). Neither PR is merged here. Each descriptor is looked up lazily, from the installed package, when report() runs. So on this branch alone, every new row reports unqualified, and it becomes provider_adapter once the package version that has the adapter is installed. The latest VM proof ran on a merge tree of this head + #697 11e6b916 + #691 66c43710 (see Testing).

  • _MEDIA_ADAPTERS adds:

    • traceai_<pkg> → <pkg>_attachment_capabilities for google_genai, vertexai, bedrock, litellm, fireworks, xai, mistralai, openrouter, portkey, ollama, vllm and minimax (groq keeps its _audio descriptor);
    • langchain, llamaindex, agno and pydantic_ai through their _attachments modules.

    together, deepseek and cerebras claim no media operation and are deliberately left out, so their rows stay unqualified. A descriptor whose operations is empty is now reported unqualified, never provider_adapter.

  • Fail-closed lookup. Each row is unqualified when:

  • Registry. auto_instrument accepts:

    • openrouter, portkey, vllm and minimax;
    • pydantic_ai / pydantic-ai (PydanticAIInstrumentor, generic keyword arguments);
    • agno (AgnoInstrumentorWrapper).
  • Agno has its own keyword arguments. Its wrapper takes OpenInference's TraceConfig and no attachment_* alias, so the generic arguments would raise TypeError. Auto passes:

    • tracer_provider;
    • attachments= the shared manager, only while capturing AND only when the installed traceai-agno declares its media descriptor. A release before the adapter has no attachments parameter; it gets the OFF arguments and keeps tracing (dd79fcfa);
    • config= an OpenInference TraceConfig carrying auto's hide_* flags, only when one is set. Without a flag, Agno keeps its own defaults and environment policy.

    With attachments OFF, Agno gets exactly tracer_provider=, plus config= when a hide flag is set. That is what configure_agno_tracing(tracer_provider=..., config=...) gets.

  • Agno fails closed on flags it cannot apply (2010273d). Agno's privacy bridge builds its FI config from the environment and takes only the hide_* flags from the OpenInference config. It has no hook for pii_redaction or hide_error_details; it reads them only from FI_PII_REDACTION and FI_HIDE_ERROR_DETAILS. So when auto's trace_config sets either flag in code, and Agno would not get it from the environment plus the forwarded hide flags (they also derive hide_error_details):

    • auto does not pass attachments= to Agno, so Agno captures no media;
    • the Agno row's status is config_unsupported (in report() and attachment_status()), with a warning when verbose;
    • Agno is still traced. Its span text does not follow that flag: PII text and error text are exported as Agno writes them. Set the environment variable to apply it.
    • The status is set with attachments ON and OFF (the text gap is the same). The keyword arguments change only with attachments ON (attachments= is dropped); OFF arguments are unchanged.
    • With the OpenAI instrumentor also on, Agno marks nothing, so traceai_openai (which does apply the flag) owns that call's media: one upload, on the OpenAI span.
  • Accurate status on a dependency conflict. BaseInstrumentor.instrument() returns without instrumenting when its requirement check fails. For example, PydanticAIInstrumentor requires the pydantic-ai distribution, and pydantic-ai-slim alone fails that check. Auto now reports such an instrumentor as dependency_unavailable, not registered.

  • Owner rule (unchanged, now proven through auto). LiteLLM over the OpenAI instrumentor, both enabled by traceai.init, uploads the call's image once, on the LiteLLM span. The OpenAI span for the nested SDK call has no refs.

  • Merged feat(python): MCP, agent SDK, smolagents and Firecrawl tool media (attachments C01/C03/C05/C06) #693 again at 92939ee9, and again at 4901f03d. The six MCP/agent descriptors are unchanged.

  • report() rows for vLLM and MiniMax. Both have been in _MEDIA_ADAPTERS, the registry and the entry points since c53acf38; the README now lists every row that can be provider_adapter (providers, frameworks, MCP/agents).

Known limits:

  • report() imports each installed instrumentor package to read its descriptor. traceai_litellm and traceai_portkey import their SDKs at module level. Set LITELLM_LOCAL_MODEL_COST_MAP=True to keep LiteLLM's import offline.
  • Agno span text follows auto's hide_* flags, but not a programmatic pii_redaction=True or hide_error_details=True (see config_unsupported above).
  • For reconciliation: feat(python): framework media capture, one owner per model call (LangChain, LlamaIndex, Agno, Pydantic AI) #691 and this branch conflict only in the module docstring of media_capture.py. Keep both paragraphs (owner mark, then per-span occurrence).

Unchanged on purpose

  • The fi-instrumentation-otel >=0.1.7 floor. The attachment import stays guarded. On a core without attachments, report() says unsupported_version, every row is unqualified, and attachment_manager= raises a clear error (subprocess test).
  • _bind and the voice bind_* helpers. They read the provider's manager, which is now also the borrowed one. The traceai/__init__.py changes in feat(python): Pipecat recorder, LiveKit owned-file and Egress import on the shared attachment core #678 (voice) touch other lines; merging the two is left to reconciliation.

Testing

67e5cbc0 (#693 4901f03d, Agno config_unsupported, auto framework+OpenAI rows).

  • New unit tests (mocked Agno wrapper, run on feat(python): one shared attachment manager in the auto module (traceai) #692 alone): a programmatic pii_redaction or hide_error_details gives no attachments= and config_unsupported; the same flag from the environment keeps attachments= and registered; hide_inputs alone (which derives hide_error_details) stays registered; OFF arguments stay exactly tracer_provider=.
  • New real-framework rows in test_auto_framework_media.py (they skip on feat(python): one shared attachment manager in the auto module (traceai) #692 alone, and run on a tree with feat(python): framework media capture, one owner per model call (LangChain, LlamaIndex, Agno, Pydantic AI) #691 and feat(python): provider media for Gemini, Vertex, Bedrock, LiteLLM and compatible chat #697): LangChain over OpenAI and Agno over OpenAI through traceai.init, in both enable orders, upload once, on the framework span; Agno with either flag only in code uploads nothing and reports config_unsupported; the environment control keeps its media; Agno + OpenAI with hide_error_details in code gives one upload, on the OpenAI span.
  • CI on 67e5cbc0: 30/30 pass. Attachments Python 3.12 has 470 passed (467 before; the 3 new unit tests ran). Provider attachments (latest) has 72 passed and 20 skipped (the 9 new framework rows skip here, as designed).
  • VM proof (2026-11-13, private GCP VM, loopback only, outbound network off for every test run). Merge tree M = this head + feat(python): provider media for Gemini, Vertex, Bedrock, LiteLLM and compatible chat #697 11e6b916 + feat(python): framework media capture, one owner per model call (LangChain, LlamaIndex, Agno, Pydantic AI) #691 66c43710 (merges only, clean on top of the earlier docstring resolution). OFF baseline BM = dd79fcfa + the same two merges.
    • Unit on this head alone (tests/attachments + traceai/tests): 470 passed on floor (3.10) and latest (3.12).
    • Report rows on M: 32/32 per-package venvs green (16 providers x floor/latest, vllm and minimax included). Each installed descriptor equals its report() row.
    • LiteLLM + OpenAI through auto: exactly one upload in both enable orders, floor and latest. The OpenAI-alone control and both OFF modes (default, kill switch) never reach F01.
    • Frameworks through auto, latest: 14 passed, 1 skipped (Pydantic AI OFF row; pydantic-ai-slim alone fails auto's dependency check, as before). All 9 rows that skip on this branch alone now ran and passed:
      • LangChain + OpenAI, both orders: one upload, on the framework span;
      • Agno + OpenAI, both orders: one upload, on the framework span;
      • Agno with pii_redaction or hide_error_details only in code: no upload, config_unsupported;
      • the same flags from the environment: media kept;
      • hide_error_details in code with OpenAI on: one upload, on the OpenAI span.
    • Floor: 12 passed, 3 skipped. The LangChain + OpenAI rows need langchain-openai, which is latest-only.
    • report() on M, latest, with hide_error_details in code: agno config_unsupported/provider_adapter; langchain and openai registered/provider_adapter; vllm and minimax not_installed/unqualified (not installed in that venv).
    • OFF parity, BM vs M, 6 groups: openai+litellm floor/latest; groq/ollama/bedrock/vllm/minimax; openai+langchain+agno floor/latest; and the same with pii_redaction in code.
      • Spans, provider requests, errors, manager type, fi- threads (none) and fork hooks are identical.
      • The three groups without Agno are byte-identical as dumped. The Agno groups differ only in Agno's per-run random values (agno.agent.id, graph.node.id, a created_at epoch), which also differ between two BM runs. With those masked they are identical.
      • The one OFF-visible change is the known one: with pii_redaction in code, the Agno row reports config_unsupported instead of registered.
    • Mutations: 7/7 detected (Agno gate removed, unit and real; status always registered; bare baseline; environment ignored, unit and real; LiteLLM owner claim). Unmutated controls pass.

dd79fcfa (Agno gate). VM, same setup:

CI on dd79fcfa: 30/30 checks pass.

c53acf38 (provider and framework media rows). All runs were on a private GCP VM, loopback only, with outbound network off for every test. There were four trees:

The only conflict was the #691 docstring in media_capture.py, resolved by keeping both paragraphs. M was never pushed.

Run Result
tests/attachments + traceai/tests, py3.10 / 3.12 456 → 466 passed (10 new), both tiers
tests/attachments_providers, floor / latest (openai/anthropic/groq CI pins) base 69p+2s / 71p; head 70p+13s / 72p+11s; M 71p+12s / 73p+10s (the new skips are SDKs not in that venv)
Report rows on M, one venv per package and tier with #697's floor and latest pins, TRACEAI_EXPECT_PROVIDER_ADAPTERS=1 plus the expected row name 32/32 green: google_genai, vertexai, bedrock, litellm, groq, fireworks, xai, mistralai, openrouter, portkey, ollama, vllm, minimax rows equal the installed descriptor (and provider_media.capabilities); together, deepseek, cerebras rows are unqualified
LiteLLM over OpenAI through traceai.init, litellm 1.53.1 + openai 1.69.0 / litellm 1.109.1 7/7 per tier: ONE upload, refs only on the completion span, the nested OpenAI span has none, in both enable orders. Control: OpenAI alone captures the same call once. OFF and kill switch: F01 never contacted
This branch alone, real packages + SDKs (litellm, google-genai, bedrock, ollama, mistralai, openrouter) every new row unqualified (no descriptor → AttributeError); openai stays provider_adapter
#697 packages on this branch's core (no provider_media) descriptor raises ModuleNotFoundError; rows unqualified
Agno through auto, #691 floor (agno 2.7.2) / latest (agno 3.1.12) image uploaded once on the model span; OFF: no F01 request and tracing unchanged; trace_config=TraceConfig(hide_inputs=True) keeps both the prompt text and the image out
Pydantic AI through auto with pydantic-ai-slim only: row dependency_unavailable, nothing traced. With pydantic-ai==2.67.0: image uploaded once; OFF: no F01 request
#691's own suite on M 81p+4s / 85p

OFF invariant. The probe ran with attachments OFF, auto-instrumented, against a loopback provider, with real calls from openai + litellm (floor and latest), groq, ollama and bedrock. M and its base export identical spans, provider requests, fi- threads (none) and fork hooks. Two base runs were also identical to each other, so the probe has no run-to-run noise. In both trees the probe's google-genai call raised RuntimeError and its mistralai import raised ImportError, the same way on both sides; those two rows prove nothing. For the six newly registered names there is no base to compare against. For openrouter, portkey, vllm and minimax, auto with attachments OFF matched calling <Instrumentor>().instrument(tracer_provider=...) directly, once object addresses in the <openai.Omit object at 0x…> reprs were normalised. The vLLM and MiniMax client checks were opened in both modes, as #697's tests do. Agno and Pydantic AI OFF behaviour is covered by the rows above.

Mutations (control: 37 unit tests and the LiteLLM rows pass). Each of these 11 made named tests fail:

  • litellm dropped from _MEDIA_ADAPTERS;
  • a wrong Agno descriptor name;
  • the empty-operations guard removed;
  • Agno given the generic arguments, caught by the unit tests and again by the real Agno run;
  • Agno hide-flag mapping removed;
  • attachments= passed to Agno while OFF;
  • the dependency-conflict status removed, caught by the unit tests and again by the real Pydantic AI run;
  • pydantic_ai registry entry removed;
  • LiteLLM's owner claim removed, which gives two uploads.

CI on c53acf38: 30/30 checks pass.

Not covered: through auto, the vLLM instrumentor traces only localhost:8000 (its default vllm_base_urls), and MiniMax traces only MiniMax hosts. auto cannot pass vllm_base_urls.

a598b9ad (MCP/agent adapters). All runs were on a private GCP VM, loopback only, with outbound network off for the all-six suite. The VM tested 42724ea6; a598b9ad changes only the CI pin (mcp<2, because mcp-use==1.7.1 needs it), and the VM used the same pin.

Suite Base 0b2eb4b5 This head
tests/attachments_providers, floor / latest 69 passed, 2 skipped / 71 passed identical
tests/attachments + traceai/tests, both tiers 446 passed 456 passed (10 new auto tests)
openai / anthropic / groq package suites, both tiers 1821, 1845 / 162, 246 / 60, 60 passed identical
New test_auto_registry.py, all six packages installed (mcp 1.30.0, mcp-use 1.7.1, openai-agents 0.24.0, claude-agent-sdk 0.2.180, smolagents 1.26.0, firecrawl-py 4.55.0, openai 3.49.0) not present 16 passed, 0 skipped
#693 MCP/agent suites, every tier (mcp, mcp-use, openai-agents floor/0.10.0/latest, claude-agent-sdk, smolagents, firecrawl) and their package suites n/a all pass; core attachments 410 passed

The openai-agents latest package suite has one collection error (InputTokensDetails validation with openai 3.x). It reproduces on #693's base and is not caused by this PR.

test_auto_registry.py runs traceai.init against the real packages:

  • each report row equals the installed descriptor's operations;
  • each of the four instrumentors enables by name with attachments OFF and ON;
  • traceai.wrap_mcp_client on a real stdio MCP session uploads the tool screenshot through the shared manager, and with attachments OFF uploads nothing and leaves the SDK result unchanged;
  • traceai.mcp_use_callback() on a real MCPAgent uploads the tool image, and uploads nothing under hide_outputs;
  • a real smolagents model-input image and a real Firecrawl inline screenshot each upload once.

In CI, TRACEAI_EXPECT_MCP_AGENT_ADAPTERS=1 turns a missing package into a failure, so a broken install cannot pass by skipping.

OFF invariant: with attachments OFF, the probe's exported span attributes and provider requests are identical to 0b2eb4b5 on both tiers.

Mutations (control: 62 passed, traceai/tests + test_auto_registry.py). Each of these 10 made named tests fail:

  • smolagents descriptor dropped;
  • mcp descriptor dropped;
  • mcp-use routed to the instrumentor path;
  • firecrawl registry key dropped (6 tests fail);
  • openai-agents entry point dropped;
  • helper drops auto's hide flags;
  • helper passes the manager while capture is off;
  • helper ignores an old package without the adapter;
  • helper never passes the shared manager;
  • wrap_mcp_client crashes on an old traceAI-mcp.

CI on a598b9ad: 30/30 checks pass, including the new auto attachments (latest) row.

0b2eb4b5 and earlier, same VM setup:
All runs were on a private GCP VM, loopback only. Real SDKs ran against a local fake provider and a TLS loopback F01 server. Floor tier: Python 3.10, openai==1.69.0, anthropic==0.41.0, groq==0.9.0. Latest tier: Python 3.12, openai==3.49.0, anthropic==2.8.0, groq==1.7.0. Base is f5e77e08, compared on the same VM.

Suite Base f5e77e08 dc69e510 0b2eb4b5 (key rename)
tests/attachments_providers, floor 63 passed, 2 skipped 69 passed, 2 skipped 69 passed, 2 skipped
tests/attachments_providers, latest 65 passed 71 passed 71 passed
tests/attachments + traceai/tests, both tiers 435 passed 444 passed 446 passed
openai package suite, floor / latest 1820 / 1844 passed, 1 failed identical to base 1821 / 1845 passed
anthropic package suite, floor / latest 162 / 246 passed identical identical
groq package suite, floor / latest 60 / 60 passed identical identical

The single openai package failure on base and dc69e510 (test_parallel_readme_responses_snippet) is a harness artefact: it reads python/frameworks/parallel/README.md, which those trimmed source tarballs left out. The 0b2eb4b5 run used the full tree, and the test passes there.

0b2eb4b5, per-call policy key renamed to the TypeScript name. media_capture.CAPTURE_POLICY_KEY and traceai._CAPTURE_POLICY_KEY are now "traceai.auto.attachment_policy.v1". Two new tests pin both keys to the TS strings and prove that a policy set under the name alone restricts capture. With attachments OFF, the probe's exported spans and provider requests are identical to dc69e510 on both tiers. Key mutations (control: 107 passed): restoring the old name in the helper only, in auto only, or in both (the original bug) each fails 5, 5 and 2 named tests. The 11 mutations below were re-run on 0b2eb4b5 (control 107 passed) and all 11 were still detected.

New end-to-end tests (tests/attachments_providers/test_auto_init.py):

  • traceai.init(attachment_config=enabled, auto_instrument=["openai", "anthropic"]) captures an OpenAI chat image and an Anthropic computer-use screenshot, end to end to F01. The screenshot keeps tool_call_id and its message position (computer_screenshot, role tool). The exported spans carry no base64. report() shows ready and provider_adapter for both integrations, with no credentials or endpoint in it.
  • A borrowed manager is shared with the instrumentors, and auto's PII flag denies on the manager itself (redaction_unavailable). Auto does not shut it down.
  • Kill switch after init: nothing is uploaded, report() says disabled, and capture resumes when the variable is unset. Kill switch at init: the old code path runs and there is not even a session bootstrap.
  • A per-call capture_content=False or muted policy, or an owner mark, uploads nothing. A permissive per-call policy still captures.

OFF invariant. The same probe ran on base and on this PR, with attachments OFF, a real OpenAI image call and a real Anthropic computer-use call. Exported span attributes and provider requests were byte-identical on both tiers. No fi- thread started. The fork hooks registered were the same as on base: one after_in_child hook from the OpenTelemetry SDK TracerProvider, and none from attachments.

Mutations: the control run passed 105 tests on dc69e510 (107 on 0b2eb4b5) (providers suite and traceai/tests, latest tier). Each of these 11 mutations made at least one named test fail:

  • the attachments= kwarg dropped: the OpenAI and Anthropic end-to-end tests and 4 others fail;
  • the borrowed manager not tightened;
  • a flag of the borrowed manager cleared;
  • a second manager created when borrowing;
  • the kill switch ignored at instrument time;
  • report() ignoring the kill switch;
  • report() media rows that are not version aware;
  • the context policy ignored;
  • a policy merge that is not deny-wins;
  • the owner mark ignored by the policy fold;
  • the owner mark ignored by Observation.

🤖 Generated with Claude Code

nik13 and others added 5 commits November 11, 2026 09:16
traceai.init passes ONE attachment manager to every instrumentor it enables,
as attachments=. It is the provider's manager, or one the application lends
with attachment_manager= (never shut down by traceai; it only gains the hide
and PII flags of trace_config). With attachments OFF, or killed by
FI_ATTACHMENTS_ENABLED=false, instrumentors receive exactly the keyword
arguments they received before.

New public helpers: get_attachment_manager(), report() (attachment state and
per-integration media support: provider_adapter, session_binding or
unqualified, version aware), with_attachment_capture_policy() and
with_attachment_ownership(). The shared provider-media helper reads the two
context keys by name: a per-call CapturePolicy can only add restrictions, and
an owner mark turns capture off for that call.

The fi-instrumentation-otel floor is unchanged; attachments stay import-guarded.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ed manager

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…edia

Shared helper fi_instrumentation.attachments.tool_media on the public
manager API and media_capture: tool media is tool output and model input,
so both directional hide flags and PII redaction deny; preflight before any
read or decode; bounded budgets; per-manager dedup; owner-key honoured.

- traceai_mcp: wrap_mcp_client() / capture_mcp_result() for typed image,
  audio and inline resource blobs (never resource URIs or links).
- traceai_mcp_use: FutureAGICallback captures typed tool blocks.
- traceai_openai_agents: computer screenshots and typed tool images/files
  on the function span, projected to fi-attachment URIs.
- traceai_claude_agent_sdk: tool_result image/PDF blocks before truncation,
  on the tool span (live or its exact ended context).
- traceai_smolagents: in-memory model-input and tool images.
- traceai_firecrawl: inline data-URI screenshots and media only.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…tions

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…mock

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
nik13 and others added 7 commits November 11, 2026 11:52
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… key

The Python auto module and the shared media helper stored the per-call
capture policy under "traceai.auto.attachment_capture_policy.v1", while the
TypeScript auto module uses "traceai.auto.attachment_policy.v1". Use the TS
name in both places so the two languages share one key name, as the owner
key already did. Tests pin both names to the TS strings and prove a policy
set by name alone restricts capture.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…nts-python-auto

Brings in the MCP, agent SDK, smolagents and Firecrawl media adapters (#693) so
the auto module can register them and report their capability descriptors.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…odule

traceai.init now enables openai_agents, claude_agent_sdk, smolagents and
firecrawl by name and hands them the one shared attachment manager, with the
same OFF keyword arguments as every other instrumentor when attachments are
off. report() reads each installed package's capability descriptor for those
four plus mcp and mcp_use, so a package version without the adapter stays
unqualified.

mcp_use has a LangChain callback, not an instrumentor: enabling it only records
it, and traceai.mcp_use_callback() builds the callback on auto's provider, hide
flags and shared manager. traceai.wrap_mcp_client(session) does the same for
MCP, whose instrumentor only propagates context. Both pass the manager only
while capture is on and the installed package has the adapter.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…eds mcp<2)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… deny mislabelled media

Claude Agent SDK: the CLI usually runs PostToolUse before it streams the tool
result. The hook ended the tool span first, so tool-result media was uploaded
against an ended span context and no span carried its fi.attachment.refs.
With attachments on, the hook now defers the span's end until its
ToolResultBlock has been captured (bounded table, 30 s timer, conversation end
and disconnect flush), then ends it at the time the hook recorded. Results
with no traced tool go to the live conversation span. With attachments off the
hook ends the span immediately, as before.

tool_media: bytes whose leading signature names another modality than the
declared MIME (a PNG labelled audio/wav or application/octet-stream) are
denied, so a mislabel cannot escape hide_input_images. OWNER_KEY aliases the
core constant when the core defines it.

smolagents: every model call that carries a captured image, new or replayed,
is marked as owner, so a provider adapter never re-uploads a replayed image.
The seen-image table gets a fresh lock in a forked child (pid check, no fork
hook).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
nik13 and others added 4 commits November 12, 2026 00:08
…k media adapters

- _MEDIA_ADAPTERS gains google_genai, vertexai, bedrock, litellm, fireworks, xai,
  mistralai, openrouter, portkey, ollama, vllm and minimax (lazy descriptors from
  provider_media), plus langchain, llamaindex, agno and pydantic_ai. together,
  deepseek and cerebras stay unclaimed, and a descriptor with no operation is
  reported unqualified.
- auto_instrument accepts openrouter, portkey, vllm, minimax, pydantic_ai and agno.
  Agno gets its own keyword arguments: no fi TraceConfig or attachment_* alias,
  auto's hide flags as an OpenInference TraceConfig, attachments only while capturing.
- An instrumentor that skipped instrumentation on a dependency conflict is reported
  dependency_unavailable instead of registered.
- Real-SDK tests: report rows equal each installed descriptor; LiteLLM over the
  OpenAI instrumentor uploads once through auto; OFF never reaches F01; Agno and
  Pydantic AI through auto.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…installed

A traceai-agno release before the adapter has no `attachments` parameter, so
auto with attachments on raised TypeError there and left Agno untraced. Auto
now passes `attachments` only while capturing AND when the installed package
declares its descriptor; otherwise Agno gets the OFF arguments.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…gate, image signatures

- Claude Agent SDK: a completed tool span waits for its result only while that
  client's receive_response() conversation is live; receive_messages() and hooks
  that run before receive_response() end the span at the hook, as before.
- Deferral tables and results-seen are per client (ToolDeferrals); a conversation
  end or disconnect() flushes only that client's waiting spans.
- tool_media: BMP, TIFF and ISO-BMFF HEIF/AVIF image brands join the signature
  table, so those images cannot escape hide_input_images under an audio, video
  or octet-stream label. Octet-stream blobs with a recognised signature are
  skipped (documented).
- Tests: receive_messages() without disconnect(), two concurrent clients,
  mislabelled BMP/TIFF/HEIF rows and the octet-stream rule.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
nik13 and others added 3 commits November 12, 2026 08:23
… cannot apply

Agno's privacy bridge reads pii_redaction and hide_error_details only from
FI_PII_REDACTION / FI_HIDE_ERROR_DETAILS. When auto's trace_config sets either
one in code and the environment (plus the forwarded hide flags) would not, auto
no longer passes attachments= to Agno and reports the row config_unsupported.
OFF arguments are unchanged. README lists the provider and framework rows,
including vllm and minimax.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…nsupported rows

LangChain and Agno over the OpenAI instrumentor through traceai.init upload
each call's image once, on the framework span (both enable orders). Agno with
pii_redaction or hide_error_details set only in code is config_unsupported and
uploads nothing; the same flag from the environment keeps its media; with the
OpenAI instrumentor also on, the unmarked call is owned by the OpenAI span.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant