Skip to content

feat(python): OpenAI, Anthropic and Groq attachments (images, files, computer-use screenshots, speech) - #690

Draft
nik13 wants to merge 4 commits into
feat/attachments-python-corefrom
feat/attachments-python-openai-anthropic
Draft

nik13 wants to merge 4 commits into
feat/attachments-python-corefrom
feat/attachments-python-openai-anthropic

Conversation

@nik13

@nik13 nik13 commented Nov 11, 2026 •

Copy link
Copy Markdown
Contributor

Part of TH-2379 (Linear).

What

Python provider attachments (design WP M01, M03, M14), stacked on the Python attachment core (#676). Mirrors the TypeScript OpenAI/Anthropic lane (#682).

  • Shared helper fi_instrumentation.attachments.media_capture (public manager API only: preflight + ticket.attach / start_stream). It is not exported from attachments.__all__.
    • Every part is preflighted before its value is read or decoded. Unused tickets are released at once. A too_large/queue_full denial is re-preflighted once at the exact decoded size.
    • Strict base64: alphabet, canonical padding, 256 KiB per inline part.
    • Deny-wins local TraceConfig folding: a hide flag for the direction, or pii_redaction, sets capture_content=False, so the core cannot relax it (not even with allow_unredacted_media_with_text_pii). hide_input_images denies input images and screenshots. hide_error_details denies is_error tool results.
    • One registry per manager of (span, position) -> occurrence: a second observer gets the first occurrence back and uploads nothing.
    • ConsumedAudio forwards only chunks the application already read. It reserves 4 MiB and steps down (2 MiB, 1 MiB, 512 KiB, 256 KiB) when the server cap is lower. Partial reads, abort and an unknown container upload nothing.
  • traceai_openai: OpenAIInstrumentor().instrument(attachments=manager) (auto's attachment_manager= is accepted too). Capture covers:
    • chat image_url and file parts;
    • Responses input_image / input_file;
    • computer_call_output screenshots (purpose computer_screenshot, role tool, tool_call_id = call_id);
    • completed image_generation_call output, non-streamed and from the consumed terminal response.completed event (partial images never);
    • images.generate b64_json;
    • speech: bytes the SDK already buffered, and the streamed body the app consumes through iter_bytes, read() or stream_to_file (the SDK's own stream_to_file is bound to the proxy, so its reads are observed). The MIME comes from container magic and is cross-checked with response_format, so a mismatch uploads nothing;
    • transcription/translation input files that the SDK holds as bytes (a path is read by the SDK itself; open file objects are never read).
    • Public: capture_openai_attachments(...), project_openai_media(...), capture_openai_speech, observe_openai_speech, capture_openai_audio_input, openai_attachment_capabilities.
  • traceai_anthropic: base64 image and PDF document blocks, and one level of tool_result content. A tool-result image is a computer_screenshot only when a typed computer_YYYYMMDD tool is declared AND an assistant tool_use of that tool has the result's id; otherwise it is tool_result. Message and block indices keep causal order for the filmstrip. An invalid tool_use_id fails closed. Public: capture_anthropic_attachments(...), project_anthropic_media(...), anthropic_attachment_capabilities.
  • traceai_groq: with an enabled manager only, new spans for audio.speech (buffered body), audio.transcriptions and audio.translations (bytes inputs), through the same helper. Without a manager nothing is wrapped.
  • Telemetry projection: content attributes are now set after capture, and only admitted media is replaced by fi-attachment:<occurrence_id> in a telemetry-only copy. The SDK request and response are never mutated.
  • Off is a strict no-op: a missing or disabled manager, or a core without attachments, leaves the instrumentors on their existing code paths. Helpers import lazily.

Explicitly unsupported (fail closed): remote URLs, provider file IDs, partial images, images.edit / variations, open file objects, speech stream_format="sse", aac/pcm/mulaw speech, beta.messages, generated images from Anthropic, Groq streamed speech and Groq path inputs.

Evidence (private GCP VM, loopback only; outbound provider traffic is a local fake server, the F01 server is TLS loopback)

  • New suite python/tests/attachments_providers: real SDKs on floor openai==1.69.0 anthropic==0.41.0 groq==0.9.0 (Python 3.10): 63 passed, 2 skipped (the skips are Groq speech, which groq 0.9.0 does not have). On latest openai==3.49.0 anthropic==2.8.0 groq==1.7.0 (Python 3.12): 65 passed. Coverage: sync and async, streaming and non-streaming, privacy negative controls (each hide flag, PII, audio-only server policy, unsampled, env disable, disabled manager), dedup, exit probes (enabled, disabled, after fork), and the real traceai.init(attachment_config=enabled) auto path.
  • Regressions, head vs base feat/attachments-python-core on the same VM, identical counts: openai 1820/1844 passed, anthropic 162/246, groq 60/60, core + auto 435/435. One openai failure, test_parallel_readme_responses_snippet, fails identically on base (pre-existing).
  • Mutations, 9 of 9 detected: read-before-preflight, PII not folded, no registry, no projection, stream_to_file unobserved, partial read finishing, any tool treated as computer, Groq speech dropped, hide_input_images ignored.

Not in this PR

🤖 Generated with Claude Code

nik13 and others added 4 commits November 11, 2026 07:47
…ared attachment manager

- fi_instrumentation.attachments.media_capture: shared, public-API-only helpers
  (preflight before any read, strict base64, deny-wins local TraceConfig folding,
  per-manager span/position registry, consumed speech audio).
- traceai_openai: chat/responses images and files, computer_call_output
  screenshots, image generation (non-stream and streamed terminal event),
  images.generate, buffered and consumed speech, transcription/translation
  bytes. instrument(attachments=...) (or auto's attachment_manager).
- traceai_anthropic: image/document blocks and tool_result media with causal
  computer-screenshot classification.
- traceai_groq: opt-in audio spans for speech and transcription/translation.
- Telemetry-only projection replaces admitted media with fi-attachment URIs.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…xported telemetry

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ackages are not installed

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…d thread order

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant