Skip to content

feat(openai,anthropic): capture typed images, documents and computer screenshots as attachments - #682

Draft
nik13 wants to merge 8 commits into
feat/attachments-ts-corefrom
feat/attachments-ts-openai-anthropic
Draft

nik13 wants to merge 8 commits into
feat/attachments-ts-corefrom
feat/attachments-ts-openai-anthropic

Conversation

@nik13

@nik13 nik13 commented Nov 11, 2026 •

Copy link
Copy Markdown
Contributor

Part of TH-2379 (Linear).

Summary

Wave B lane M02/M04: typed media capture for @traceai/openai and @traceai/anthropic through the shared AttachmentManager from @traceai/fi-core/attachments (stacked on #677).

  • Constructors accept attachments: provider.attachments. With no manager, or a disabled one, span telemetry is unchanged.
  • OpenAI capture covers:
    • Chat input image_url and file parts;
    • Responses input input_image and input_file parts, plus computer_call_output screenshots (purpose computer_screenshot, linked by call_id);
    • Responses image_generation_call output, non-streamed, and streamed from the terminal response.completed event the application consumed;
    • images.generate b64_json output (a new OpenAI Images span).
  • Anthropic capture covers Messages image and document base64 blocks, and one level of tool_result content. Screenshots are classified only when a typed computer_YYYYMMDD tool and an assistant tool_use name it.
  • Every part is preflighted with the core's manager.preflight(...) (deny-wins, budget reserved) before its value is read or decoded, then committed with ticket.attach(...). Only inline data is supported, capped at 256 KiB per part. Base64 must be strict (alphabet, padding, round-trip). No URL or file fetching.
  • Telemetry projection: only admitted parts are replaced, by a fi-attachment:<occurrence_id> URI. Denied or not-captured parts keep today's payload. The request sent to the SDK is never mutated.
  • Dedup: a second observer (CJS/ESM copy, or an SDK plus wire observer) of the same span and position reuses the first occurrence. It does not upload again.
  • Public helpers: captureOpenAIAttachments(manager, spanOrCorrelation, payload, {operation, direction, traceConfig?, capture?}) and captureAnthropicAttachments(manager, spanOrCorrelation, payload, {traceConfig?, capture?}). Both return the admitted positions. Also exported: projectOpenAIMedia, projectAnthropicMedia, and frozen capability descriptors.

Privacy

  • Deny wins. A local instrumentor traceConfig (hideInputs/hideInputMessages/hideInputText, hideOutputs/..., piiRedaction, hideInputImages, hideErrorDetails for is_error tool results) adds captureContent:false, on top of the manager's own policy. So the core's allowUnredactedMediaWithTextPii opt-in cannot relax a local instrumentor's piiRedaction. Python M01/M03 should match this.
  • A denied source is never read: the part's value is not even accessed, and nothing goes on the network. Error bodies are never treated as generated media.

Explicitly unsupported (fails closed, named in the descriptors)

  • Both packages: remote URLs and provider file ids.
  • OpenAI: partial (preview) images, streamed images.generate, images.edit and images.createVariation, and speech (lane M15).
  • Anthropic: SDKs with native OpenTelemetry (left unpatched, as before), generated images, and beta.messages.

Behaviour notes

  • Content attributes (input messages, raw input, invocation parameters) are now set right after span start instead of in the start options. This lets admitted media be projected out first. Kind, model, provider and operation are still set at start. FISpan masks setAttributes the same way.
  • Once U01 lands, captured images show in the trace view as fi-attachment: URIs.

Testing

  • typescript/packages/traceai_openai/test/attachments-providers.test.cjs runs real SDKs against loopback provider and attachment-API mocks. It covers:
    • bytes and causal positions;
    • every privacy flag, server redaction and metadata policy;
    • integration restrictions;
    • malformed, oversized and hostile payloads;
    • unsampled spans and queued policy tightening;
    • CJS/ESM dedup;
    • streamed Responses (the consumed terminal event's image uploads once and is projected out; a consumer that stops early uploads nothing);
    • raw asResponse() callers;
    • disabled-manager telemetry staying byte-identical.
  • New workflow openai-anthropic-attachments-tests.yml runs it network-isolated for OpenAI 4.0.0, 5.23.2, 6.0.0, 7.47.0 and latest, paired with Anthropic 0.20.0, 0.27.3, 0.133.0, 0.134.0 and latest. It also runs both packages' jest suites.
  • The existing OpenAI SDK contracts and Anthropic TypeScript tests workflows were dispatched on this branch, because their pull_request trigger only fires for dev/main bases.
  • GCP VM run (Node 22.23.4, head 4b52c19, all suites network-isolated): OVERALL_FAIL=0.
    • Attachment suite: 44/44 for each pair. OpenAI 4.0.0/5.23.2/6.0.0/7.0.0/7.47.0/7.55.0 (latest) were paired with Anthropic 0.20.0/0.27.3/0.133.0/0.134.0/1.8.0 (latest).
    • Jest: openai 304 passed (2 skipped); anthropic 82 passed (3 skipped).
    • OpenAI currentSdk (json and proto+gzip, manual), recall/privacy regressions and apiPromise pass for 4.0.0, 7.47.0 and latest.
    • Anthropic native-boundary, release-lifecycle (normal and privacy) and README pass for all 5 pinned SDKs.
    • fi-core attachment suites pass.
  • Negative controls: 10 of 10 detected. Each one removes a single rule from the built adapter, and the suite then fails:
    • local hide flags (both packages);
    • hideInputImages (both packages);
    • the hideErrorDetails error-detail flag;
    • output error bodies;
    • the projection;
    • dedup;
    • the 256 KiB inline cap (both packages).

2026-11-11: resync with the updated core (#677 at f0a56b3)

  • Merged feat/attachments-ts-core (f0a56b3) into this branch (merge, no rebase). The merge was clean, and the unchanged lane already passed 44/44 on it, so capture kept working.
  • Adopted the core preflight API (SDK streaming design, section 1) in place of lazy get data() getters:
    • each part is preflighted at the 256 KiB inline cap before its value is read;
    • an admitted ticket that is not used (malformed base64, a MIME outside the allowlist, decode failure) is released at once, not left to the ticket TTL;
    • an OpenAI data URI naming a sibling MIME of the same modality (for example image/jpeg after an image/png preflight) is released and re-admitted under its exact MIME before decoding;
    • if the server negotiates an upload limit below 256 KiB (too_large), or the owned-byte budget is nearly used (queue_full), the part is re-preflighted once with its exact decoded length. Preflight reaches either answer only after every privacy, correlation and sampling step has passed. So small media keeps capturing exactly where the old attach(), which reserved only the actual length, captured it.
  • Streamed Responses output: the generated image in the terminal response.completed event is captured and projected out of the span output before the span ends. This uses the iterator the application already consumes, with no extra reads, clones or tees. It is a complete base64 value, so it goes through preflight and attach, not the byte-stream API. Partial images stay unsupported.
  • Capability descriptor: responses.stream.output added to operations. streamed_output was removed from unsupported, and images.generate.stream was added.
  • Test fixtures now mint one-hour sessions, matching the core's renewal window. They assert no FI_ATTACHMENTS_* degradation warning and no leaked tickets or bytes. New rows cover a negotiated limit, a nearly used budget, and a sibling MIME.
  • GCP VM run on head 88a4c23 (Node 22.23.4 and 18.20.8, network-isolated):
    • the attachment suite passes 50/50 for OpenAI 4.0.0/5.23.2/6.0.0/7.47.0/7.55.0 paired with Anthropic 0.20.0/0.27.3/0.133.0/0.134.0/1.8.0, with 0 core warnings;
    • jest: openai 304 passed (2 skipped), anthropic 82 passed (3 skipped);
    • the lockfile is unchanged.
  • Attribution:
    • the new test file against the merged-but-unchanged build fails exactly the descriptor row and the streamed-capture row (48/50);
    • against 912c4fb (before the queue_full retry) it fails exactly the two nearly-used-budget rows.
  • Negative controls (6 of 6 detected):
    • dropping the ticket release fails 2 rows;
    • reading the value before preflight fails 17 rows;
    • restoring the streamed-path gate fails the real-SDK row;
    • dropping the exact-length retry fails 4 rows;
    • retrying on too_large only fails the 2 budget rows;
    • dropping the sibling-MIME re-admit fails its row.

Not in this PR

  • Real product playback and the computer-use filmstrip. They need the mounted backend gateway (reconciliation item 3) and UI U01/U03 (Wave C).

🤖 Generated with Claude Code

nik13 and others added 5 commits November 10, 2026 19:59
… attachment manager

OpenAI chat/responses input images, files and computer_call_output
screenshots, Responses image_generation_call and Images generate b64
outputs; Anthropic Messages image/document blocks and nested tool_result
media with computer-tool screenshot classification. Decode is lazy behind
manager admission, inline only and capped at 256 KiB per part. Telemetry
payloads are projected so media bytes and filenames never reach OTLP.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…telemetry otherwise

Capture now runs before the telemetry projection and returns the admitted
positions. Only those parts are replaced by their fi-attachment occurrence
URI; denied, disabled or absent capture leaves the span payload exactly as
before. A second observer of the same span position reuses the first
occurrence. Streamed Responses output media and the Images edit and
variation endpoints are explicitly unsupported. Adds a network-isolated
CI workflow against real SDK floor and latest releases.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…dently of the manager

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…r SDK typings

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…d data URIs

The cloud mutation run showed the OpenAI cap was not exercised: the oversized
case was not a data URI, so it failed the prefix check first.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
nik13 and others added 3 commits November 11, 2026 02:35
…nai-anthropic

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…Responses output

Adopt the core preflight API (sdk-streaming-design section 1) in place of
lazy `get data()` getters:

- Each inline part is preflighted (deny-wins, budget reserved) before its
  value is read or decoded. Denied parts are never read.
- An admitted ticket that is not used (malformed base64, MIME outside the
  allowlist, decode failure) is released at once, not left to the TTL sweep.
- OpenAI data URIs naming a sibling MIME of the same modality are released
  and re-admitted under the exact MIME before decoding.
- When the server negotiates an upload limit below the 256 KiB inline cap,
  preflight returns too_large only after every privacy step passed, so the
  part is re-preflighted with its exact decoded length. Small media keeps
  capturing as before.

Streamed Responses: the generated image in the terminal response.completed
event that the application consumed from its own iterator is now captured
and projected out of the span output, before the span ends. Partial
(preview) images and streamed images.generate stay unsupported and are
named in the capability descriptor.

Test fixtures mint one-hour sessions (core renewal window), assert no
FI_ATTACHMENTS_* degradation warnings and no leaked tickets or bytes.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ct size

A cap-sized (256 KiB) preflight can return queue_full when the owned-byte
budget is nearly used, where the previous attach() reserved only the actual
length and admitted small media. queue_full, like too_large, is evaluated
after every privacy, correlation and sampling step, so the part is sized
exactly and preflighted once more. New test row for both providers.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant