Repository navigation
Conversation
…edia Shared helper fi_instrumentation.attachments.tool_media on the public manager API and media_capture: tool media is tool output and model input, so both directional hide flags and PII redaction deny; preflight before any read or decode; bounded budgets; per-manager dedup; owner-key honoured. - traceai_mcp: wrap_mcp_client() / capture_mcp_result() for typed image, audio and inline resource blobs (never resource URIs or links). - traceai_mcp_use: FutureAGICallback captures typed tool blocks. - traceai_openai_agents: computer screenshots and typed tool images/files on the function span, projected to fi-attachment URIs. - traceai_claude_agent_sdk: tool_result image/PDF blocks before truncation, on the tool span (live or its exact ended context). - traceai_smolagents: in-memory model-input and tool images. - traceai_firecrawl: inline data-URI screenshots and media only. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…tions Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…mock Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Contributor
Author
Qualification (private cloud VM, loopback only)The new suite
🤖 Generated with Claude Code |
nik13
added a commit
that referenced
this pull request
Nov 11, 2026
…nts-python-auto Brings in the MCP, agent SDK, smolagents and Firecrawl media adapters (#693) so the auto module can register them and report their capability descriptors. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… deny mislabelled media Claude Agent SDK: the CLI usually runs PostToolUse before it streams the tool result. The hook ended the tool span first, so tool-result media was uploaded against an ended span context and no span carried its fi.attachment.refs. With attachments on, the hook now defers the span's end until its ToolResultBlock has been captured (bounded table, 30 s timer, conversation end and disconnect flush), then ends it at the time the hook recorded. Results with no traced tool go to the live conversation span. With attachments off the hook ends the span immediately, as before. tool_media: bytes whose leading signature names another modality than the declared MIME (a PNG labelled audio/wav or application/octet-stream) are denied, so a mislabel cannot escape hide_input_images. OWNER_KEY aliases the core constant when the core defines it. smolagents: every model call that carries a captured image, new or replayed, is marked as owner, so a provider adapter never re-uploads a replayed image. The seen-image table gets a fresh lock in a forked child (pid check, no fork hook). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…gate, image signatures - Claude Agent SDK: a completed tool span waits for its result only while that client's receive_response() conversation is live; receive_messages() and hooks that run before receive_response() end the span at the hook, as before. - Deferral tables and results-seen are per client (ToolDeferrals); a conversation end or disconnect() flushes only that client's waiting spans. - tool_media: BMP, TIFF and ISO-BMFF HEIF/AVIF image brands join the signature table, so those images cannot escape hide_input_images under an audio, video or octet-stream label. Octet-stream blobs with a recognised signature are skipped (documented). - Tests: receive_messages() without disconnect(), two concurrent clients, mislabelled BMP/TIFF/HEIF rows and the octet-stream rule. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Part of TH-2379 (Linear).
What
Tool media capture for the Python MCP and agent integrations (attachments WP C01, C03, C05, C06). It mirrors the TypeScript MCP and Agent SDK lane (#681) and reuses the shared helper
fi_instrumentation.attachments.media_capturefrom #690.Base: this PR is stacked on
feat/attachments-python-openai-anthropic(#690), not directly onfeat/attachments-python-core, because it builds on #690's sharedmedia_capturehelper rather than forking it. When #690 lands in the core branch, this PR can be retargeted.Shared helper:
fi_instrumentation.attachments.tool_media(not in__all__)hide_inputs,hide_outputs, and their message and text variants),hide_input_imagesfor images,hide_error_detailsfor error results, andpii_redactionall deny capture. The core cannot relax any of them.manager.preflightadmits it. Each value is read once. Base64 is decoded strictly.traceai.auto.attachment_owner.v1captures nothing here. This uses the same key name as the TypeScript lanes and feat(python): one shared attachment manager in the auto module (traceai) #692;tool_media.OWNER_KEYaliasesmedia_capture.OWNER_KEYwhere the core defines it. smolagents sets this mark around every model call that carries a captured image, new or replayed. Provider adapters honour the mark only once feat(python): one shared attachment manager in the auto module (traceai) #692'smedia_capturelands, so until then OpenAI + smolagents on one manager upload such images twice.audio/wav), the part is denied, so a mislabel cannot escapehide_input_imagesor a modality restriction. Nothing is relabelled.tool_media._SIGNATURES): PNG, JPEG, GIF, WebP, BMP (BM), TIFF (II*\0,MM\0*), ISO-BMFFftypwith the image brandsavif,heic,heix,mif1andmsf1, WAV, Ogg, FLAC, MP3 with an ID3 tag, and PDF.ftypbrands (MP4, M4A, QuickTime), WebM/Matroska, MP3 without an ID3 tag, and any image format not listed above.application/octet-streamis the genericfilemodality, so an octet-stream blob whose bytes carry one of the image, audio or PDF signatures above is skipped. This is a deliberate fail-closed over-deny: for example a PDF sent as an octet-stream blob is not uploaded.Integrations
traceai_mcp:wrap_mcp_client(session, tracer_provider=..., attachments=None, trace_config=None, attachment_capture=None)andcapture_mcp_result(manager, span, result, operation).tracer_provider.attachment_manager.call_toolorread_resourcegets anmcp.tools/callormcp.resources/readCLIENT span.ImageContent,AudioContent, andBlobResourceContents.blob(embedded or fromresources/read).resource_link, resource URIs,structuredContent, and MIME types relabeled to another modality.MCPInstrumentorstays propagation-only.traceai_mcp_use:FutureAGICallback(attachments=..., attachment_capture=...)captures the typedimageandfileblocks that mcp-use 1.7 builds, with the tool call id. LangChain 1.x turns audio tool content into a string, so audio is not captured.traceai_openai_agents: captures on the function span:computer_screenshot(the SDK records these on a function span from 0.10.0);ToolOutputImageandToolOutputFileContentdata URIs (0.4.0 and later).Admitted values become
fi-attachment:<occurrence_id>inoutput.value. Remote URLs and file IDs are never fetched.trace_include_sensitive_data=Falsemeans nothing is recorded or uploaded.traceai_claude_agent_sdk:ToolResultBlockbase64 image and PDF blocks are uploaded from the original SDK message, before any text flattening or truncation. They attach to the tool span and are projected on it (fi.attachment.refs) in both stream orders. The CLI usually runs PostToolUse before it streams the tool result. So with attachments on, and only while that client'sreceive_response()conversation span is live, the hook keeps the completed span recording until its result has been captured, then ends it at the time the hook recorded. The wait is bounded: at most 256 spans per client, a 30 s timer, and a flush when that conversation ends and on that client'sdisconnect(). Each client has its own waiting table and results-seen table (ToolDeferrals), so one conversation ending never ends another client's waiting span. Withreceive_messages()(tool media is captured fromreceive_response()only), or for a hook that runs beforereceive_response()starts, the hook ends the span immediately, as it always does with attachments off. A result with no recording tool span goes to the live conversation span.tool.outputtext and the conversation output text. Before this change, base64 could reach both through[:100]on a list and the 5000-character JSON dump.traceai_smolagents: captures images handed to a model (task images andobservations_images) on the model span, once per image object, and in-memory tool images on the tool span. A replayed image is not uploaded again, and its model call is still marked as owner.AgentAudioand path-backed images are never read. In a forked child, the seen-image table starts with a fresh lock (pid check, no fork hook).traceai_firecrawl: captures Documentscreenshot,actions.screenshots,audioandvideoonly when they are inlinedata:URIs. Hosted URLs (what Firecrawl Cloud returns) are never fetched. Tested onscrape(sync and async) andget_crawl_status;crawl,batch_scrapeand the batch status calls use the same Document reader but are not separately tested.Each instrumentor accepts
attachments=(and auto'sattachment_manager=). With no manager, a disabled manager, or a core without attachments, the old code paths run unchanged.Behaviour change with attachments off (intentional)
In
traceai_claude_agent_sdk,tool.outputand the conversationoutputtext no longer carry inline media bytes, even without a manager. Before this change, base64 could reach both through[:100]on a list and the 5000-character JSON dump. This fixes an existing privacy leak. Everything else is unchanged without a manager.Not done (fails closed)
RemoteUrlretrieval, so these are never fetched and nothing is uploaded for them (source_unavailable). Only inlinedata:URIs are captured. C06 is therefore met only for self-hosted or inline results, not for Firecrawl Cloud.media_capture.REGISTRY(from feat(python): OpenAI, Anthropic and Groq attachments (images, files, computer-use screenshots, speech) #690) uses the same lock-plus-table shape as the smolagents seen table, without a fork reset. That file belongs to feat(python): OpenAI, Anthropic and Groq attachments (images, files, computer-use screenshots, speech) #690 and is not changed here._active_tool_spans(spans between PreToolUse and PostToolUse) and the tool-span parent are still module-global. A conversation that ends while another client's tool is still running ends that tool span with an error status, and concurrent conversations can parent tool spans on each other's conversation. Reusing oneClaudeAgentOptionsobject for two clients injects the hooks twice. Only the deferral tables are per client.receive_response()(for examplequery(), other work, thenreceive_response()while a fast tool already finished), the tool span ends at the hook and its tool-result media lands on the conversation span. With a real model call betweenquery()and the tool, the client is already iterating.Review fixes (d8ee8cd, 92939ee)
fi.attachment.refs. Ingest now binds on the upload's authenticated parent plus a live span row, so those occurrences were not unbindable. They were missing from the F01 per-span projection. Fix: the span ends late, at the time the hook recorded (above). The_endedcontext table is removed.AudioContentlabelledaudio/wavcarrying PNG bytes was uploaded underhide_input_images=True. Fix: the signature check above.isErrorresult underhide_error_detailshad no test. Added.Second review fixes (4901f03)
receive_response()(P2). On 92939ee, a client streaming withreceive_messages()and never callingdisconnect()exported no tool span: it stayed parked until a timer that never fired. Fix: the hook defers only while that client'sreceive_response()conversation span is live. New rows:receive_messages()withoutdisconnect(), and a conversation end that flushes its own waiting span with nodisconnect().ToolDeferrals; a conversation end ordisconnect()flushes only that client's entries. New row: two clients, B's tool span waits while A runs to completion, then B's media lands on B's tool span.audio/wav,audio/L16,audio/mp4,video/mp4orapplication/octet-streamwere uploaded underhide_input_images=True. Fix: those signatures were added to_SIGNATURES. The wording in this body and in the MCP README now lists exactly the signatures checked.tool_mediadocstring and above: an octet-stream blob carrying a recognised image, audio or PDF signature is skipped.Tests
New suite
python/tests/attachments_mcp_agentsand workflowattachments-python-mcp-agents-tests.yml. Each tier uses real SDK releases against a TLS loopback F01 server and loopback doubles (an MCP stdio server, a scripted AgentsModel, a stream-json Claude CLI stand-in, an OpenAI-compatible API, and a Firecrawl API), with outbound network disabled.VM results, regression counts against the base and the mutation results for be331c3 are in the comment below.
Review-fix run (GCP VM, private, deleted after the run), head d8ee8cd plus 92939ee:
tests/attachments: 410 passed.audio/wavunderhide_input_images: uploaded on be331c3, denied on head.[true, null]on be331c3,[true, true]on head.disconnect()also flushes the waiting spans;data:prefix guard, because the structural parse already rejects non-data:strings, so that mutation changes no behaviour.GitHub Actions on d8ee8cd: all 29 checks pass, including the 14-job MCP/agent matrix.
Second review-fix run (GCP VM, private, deleted after the run), head 4901f03 vs base 92939ee:
tests/attachments: 410 passed. Package suites for mcp, claude-agent-sdk and smolagents, floor and latest: head and 92939ee give identical results, 0 failures each.hide_input_images: BMP asaudio/wav, BMP asaudio/L16, BMP as an octet-stream blob and TIFF asaudio/L16were uploaded on 92939ee and are denied on head. Honest WAV, MP3 without ID3, PDF and PNG still upload; a PDF octet-stream blob is skipped on both (the documented rule).avif/mif1/msf1asvideo/mp4) also pass on 92939ee, because the loopback policy does not admit video; theirsignature_kindrows fail there.livenever decremented, and removing BMP, TIFF,avifor the other HEIF brands.GitHub Actions on 4901f03: all 29 checks pass, including the 14-job MCP/agent matrix.
🤖 Generated with Claude Code