Repository navigation
feat(auto): attachments in @traceai/auto (A02 ownership, A03 wire JSON/FormData, A04 consumed audio) - #687
Draft
nik13 wants to merge 23 commits into
Draft
feat(auto): attachments in @traceai/auto (A02 ownership, A03 wire JSON/FormData, A04 consumed audio)#687nik13 wants to merge 23 commits into
nik13 wants to merge 23 commits into
Conversation
… attachment manager OpenAI chat/responses input images, files and computer_call_output screenshots, Responses image_generation_call and Images generate b64 outputs; Anthropic Messages image/document blocks and nested tool_result media with computer-tool screenshot classification. Decode is lazy behind manager admission, inline only and capped at 256 KiB per part. Telemetry payloads are projected so media bytes and filenames never reach OTLP. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…telemetry otherwise Capture now runs before the telemetry projection and returns the admitted positions. Only those parts are replaced by their fi-attachment occurrence URI; denied, disabled or absent capture leaves the span payload exactly as before. A second observer of the same span position reuses the first occurrence. Streamed Responses output media and the Images edit and variation endpoints are explicitly unsupported. Adds a network-isolated CI workflow against real SDK floor and latest releases. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…dently of the manager Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…r SDK typings Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…d data URIs The cloud mutation run showed the OpenAI cap was not exercised: the oversized case was not a data URI, so it failed the prefix check first. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…nai-anthropic Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…Responses output Adopt the core preflight API (sdk-streaming-design section 1) in place of lazy `get data()` getters: - Each inline part is preflighted (deny-wins, budget reserved) before its value is read or decoded. Denied parts are never read. - An admitted ticket that is not used (malformed base64, MIME outside the allowlist, decode failure) is released at once, not left to the TTL sweep. - OpenAI data URIs naming a sibling MIME of the same modality are released and re-admitted under the exact MIME before decoding. - When the server negotiates an upload limit below the 256 KiB inline cap, preflight returns too_large only after every privacy step passed, so the part is re-preflighted with its exact decoded length. Small media keeps capturing as before. Streamed Responses: the generated image in the terminal response.completed event that the application consumed from its own iterator is now captured and projected out of the span output, before the span ends. Partial (preview) images and streamed images.generate stay unsupported and are named in the capability descriptor. Test fixtures mint one-hour sessions (core renewal window), assert no FI_ATTACHMENTS_* degradation warnings and no leaked tickets or bytes. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ct size A cap-sized (256 KiB) preflight can return queue_full when the owned-byte budget is nearly used, where the previous attach() reserved only the actual length and admitted small media. queue_full, like too_large, is evaluated after every privacy, correlation and sampling step, so the part is sized exactly and preflighted once more. New test row for both providers. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ream capture A02: pass the shared manager to instrumentors as `attachments` (the option every adapter lane uses), keep media status apart from span status, and stop disabling a borrowed manager on an env override (updatePolicy is tighten-only; the core applies the override at admission). With attachments OFF, wire skip/inspection is unchanged. A03: every FormData Blob and speech buffer is preflighted at its exact size before it is read; JSON media (chat/responses inputs, image outputs, computer screenshots) goes through the @traceai/openai adapter so SDK and wire observations of one span share an occurrence. Wire-only JSON input uses the wire path's own bounded request copy. A04: chunks the application reads (reader read() or async iterator next()) feed startAttachmentStream under a native model span with an explicit capture policy. Cancel, early return, pipe, tee and clone abort; nothing partial is uploaded. Wire-only incremental reads and wire-only FormData stay unsupported until the core can preflight a late-bound target. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… fix test doubles The latest OpenAI SDK (7.x) consumes JSON with text(), so wire-only image output is now read from the wire path's own bounded response copy at bind, whatever method the application used. Test doubles read Blob request bodies; T-CS-01 ignores undici's own request-body tee and reads. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…s exports map Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This was referenced Nov 11, 2026
… deferred targets Merges feat/attachments-ts-core 930876e and adopts its deferred-target preflight in the wire media observer: - Wire-only FormData audio (OpenAI transcriptions/translations, the openai 5.x+ SDK route) and wire-only incremental speech reads (reader or async iterator; OpenAI speech, Deepgram speak) are admitted against a deferred target before the wire span exists. The core holds bounded bytes, reads and sends nothing, and re-checks policy, environment and sampling when the wire span binds. - Every path on which no wire span will be bound (unmatched route, sync or async failure, emitter delivery gate, build failure, invalid span identity) cancels the target at once; the 300 s bind deadline is only the backstop. A late bind returning false is treated as terminal. - JSON media is unchanged: captured at bind from the wire path's own bounded copies, never through a deferred target. - report().integrations[].media reports provider_adapter for anthropic (version aware, like openai). Tests: wire-only FormData and reader/iterator rows flipped from "not captured" to one upload bound to the exported wire span; no-span, unsampled, image-edit (no wire span) and policy-tightened-while-held rows; deferred path inside the privacy matrix; the env-override test now builds a process-env manager and proves the kill switch at admission; new child-process rows drive auto.init() with an enabled manager against allowlisted HTTPS hosts (and with the kill switch flipped after bootstrap), and the installed @traceai/anthropic instrumentor end to end. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… at once Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…wire-only materialized refs now project onto the wire span Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ture A fetch that rejected (connection drop, AbortSignal abort) or whose response body failed while read still emits an error wire span, but the deferred target held for wire-only FormData audio or incremental speech is no longer bound to it: emit gets no media binder, and settle() cancels the target at once, as on the synchronous-throw path. A call that completed with an HTTP error status is a finished call and still binds. Regression rows: socket reset, abort after send and a response body that fails mid-read each upload nothing and free the held bytes; a completed 400 binds the held audio to its wire span. CONFIGURATION.md and the settle() comment now state the exact boundary. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…rors come from Node's realm) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
One call seen by two layers: an adapter that captures claims the call with
Symbol.for("traceai.auto.attachment_owner.v1") around the provider SDK call,
and the wire layer then uploads nothing for that call. This adds the rows
the suite did not have: wire-only incremental reads (reader and async
iterator, now captured through a deferred target) and FormData under the
claim, for the OpenAI speech and Deepgram speak routes, with and without an
outer capture policy, an outer owner with no adapter upload, a concurrent
owned and unowned call (the claim does not leak), and the real OpenAI SDK.
The provider-media claim is stubbed through the public Symbol key, so the
file needs no adapter package.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Part of TH-2379 (Linear).
What this does
Brings attachments (images, files, computer-use screenshots and speech audio) into the zero-code agent,
@traceai/auto. Work packages A02, A03 and A04 of the attachments design. Attachments stay off by default: with the default disabled manager, auto's wire skip and inspection decisions are exactly the ones it made before this change.This branch includes #682 (OpenAI/Anthropic attachment adapters, merged in, not rebased) because auto's JSON media capture calls that lane's
captureOpenAIAttachments, so an SDK span and a wire observation of the same span and position share one occurrence. Once #682 lands onfeat/attachments-ts-core, this diff shrinks totypescript/packages/traceai_auto/**.A02: one manager, ownership and status
attachmentManager(borrowed), else the provider'sattachments, else the one its ownregister()creates.getAttachmentManager()returns it.attachments, the option name every adapter lane uses (feat(frameworks): typed framework media attachments (LangChain, LlamaIndex, Vercel AI SDK, Genkit, Mastra) #679, feat(coding-agents): opt-in typed image attachments for coding-agent integrations #680, feat(mcp,claude-agent-sdk): capture typed MCP tool media and original Agent SDK media #681, feat(openai,anthropic): capture typed images, documents and computer screenshots as attachments #682, feat(providers): typed provider media capture for TypeScript Gemini, Bedrock, compatible chat and speech #683). Proven end to end: auto + the real OpenAI SDK +@traceai/openaiuploads the image once, bound to the native LLM span.FI_ATTACHMENTS_ENABLED=false(updatePolicyis tighten-only). The core applies that override at every admission for a manager whoseenvironmentreadsprocess.env(auto's own manager and thecreateNodeAttachmentManagerdefault do); a borrowed manager built with a customenvironmentsees only what that function returns.report().attachmentsgives manager state, queue counters, allowlisted routes, sources and unsupported sources.report().integrations[].media(provider_adapterforopenaiandanthropic/unqualified) is kept apart from span status and is version aware (an installed instrumentor without the capability descriptor reportsunqualified).A03: allowlisted wire JSON and FormData media
manager.preflightat its exact size BEFORE it is read, copied or decoded (no moreget data()getters).tool_call_id, imagesb64_json, responsesimage_generation_call) goes through the@traceai/openaiadapter only: no private parser, no recursive scraping, no URL fetch.json(),text()orarrayBuffer()(openai 7.x usestext()).A04: consumed wire audio, no eager reads
arrayBuffer()/blob()) is captured, bound to the actual wire span.read()or async iteratornext()) feedstartAttachmentStream. Cancel, earlyreturn, pipe, tee and over-cap objects upload nothing; nothing partial is ever uploaded as complete.Symbol.for("traceai.auto.attachment_owner.v1"), set by the ElevenLabs adapter in feat(voice): TS LiveKit Agents recording, Pipecat recording link, ElevenLabs speech attachments #685) is honoured.Additions in
60ef0c90(2026-11-12): deferred targets adopted, review notes fixedMerges
feat/attachments-ts-coreat930876ea(deferred-target preflight, late/unsampled bind refusal) by merge, not rebase.Wire-only FormData and wire-only incremental speech are now captured (previously
correlation_unavailable):openai5.x+ SDKaudio.transcriptionsroute) and wire-only incremental speech reads (reader or async iterator; OpenAI speech and the Deepgram SDKspeakroute) are admitted against a deferred target before the wire span exists. Same deny-wins admission as before; the core holds bounded bytes (at most half the owned budget) and reads or sends nothing.association.parent, and the reference is projected onto it).bind()returningfalse, which also covers a late bind, is treated as terminal). The bind deadline is the core maximum, 300 s, because a long transcription can take minutes; it is only the backstop (for example a hook removed mid-flight).Review notes:
report().integrations[].mediareportsprovider_adapterforanthropic(version aware, likeopenai).auto.init()with an ENABLED manager against allowlisted HTTPS hosts (api.openai.comspeech and transcription,api.deepgram.comspeak): each call uploads once, bound to a distinct exported wire span. The same calls withFI_ATTACHMENTS_ENABLED=falseset after bootstrap send no attachment traffic and leave provider traffic and wire spans unchanged.environmentreadsprocess.env: it proves the kill switch denies at admission (native and wire-only, materialized, incremental and FormData) and that lifting it resumes capture on the same borrowed manager.@traceai/anthropicunderauto.init(): one image upload bound to the Anthropic LLM span.process.env).Testing for
60ef0c90(private GCP VM, deleted after the run; tests in a loopback-only network namespace; synthetic media and keys). Control =04e3e5d9(this branch's previous head merged with ts-core930876ea, no lane change).attachments,attachment-lifecycle,attachment-adapters), Node 22.23.4 / 20.19.5 / 24.11.1openaiSDK rows x 4.52.7 / 5.23.2 / 6.0.0 / 7.55.0@traceai/autosuite, Node 22settle(3 rows), no deferred bind (10),bind()==falseignored (1), no deferred target (12), anthropic media flag removed (2), env disables a borrowed manager (1), nosettleafteremit(1)GitHub Actions on
60ef0c90:@traceai/autosuite and the OpenAI/Anthropic attachment matrix (OpenAI 4.0.0 to latest, Anthropic 0.20.0 to latest) all pass.Fix in
1afa12d6(2026-11-12): a failed or aborted call drops the held captureReview finding: the
60ef0c90text above said a sync or async fetch failure cancels the held wire-only capture, but only the synchronous-throw path did. A rejected fetch (connection drop,AbortSignalabort after the send) emitted the error wire span with the media binder, so held FormData audio uploaded bound to that ERROR span.fetch-runtime.tscomplete(): whenresponse.erroris set,emitgets no media binder, andsettle()cancels the deferred target at once. This now matches the synchronous-throw path.response.errorcovers a rejected fetch (network error or abort) and a response body that fails or is aborted while it is read.docs/CONFIGURATION.mdand thesettle()comment now state this boundary exactly.attachments.test.ts:Testing for
1afa12d6: two private GCP VMs, both deleted after the run. Tests ran in a loopback-only network namespace with synthetic media and keys. The VM tree was60ef0c90plus this diff (byte-identical patch).attachments,attachment-lifecycle,attachment-adapters), Node 22.23.4 / 20.19.5@traceai/autosuite, Node 22 (first VM, at9e0a44a9)toBeInstanceOf(Error)assertion.1afa12d6fixes the assertion, and those rows pass in the run aboveauto.init(), a process-env manager and a loopback providerGitHub Actions on
1afa12d6: see the checks on this PR.Test added in
93a2aa46(2026-11-12): one call, two layers, one uploadtraceai_auto/test/attachment-owner.test.ts(test only; no source change). It pins the owner rule for one call seen by two layers: an adapter that captures claims the call withSymbol.for("traceai.auto.attachment_owner.v1")around the provider SDK call, and the wire layer then uploads nothing for that call. Before this, every owner row in the suite used a native span witharrayBuffer(); none covered wire-only incremental reads, which60ef0c90started capturing through a deferred target.arrayBuffer(), each with and without an outer capture policy. The body is read after the claim scope has ended, as an SDK caller does.openaiSDK.@traceai/provider-media, feat(providers): typed provider media capture for TypeScript Gemini, Bedrock, compatible chat and speech #683) is stubbed through the public Symbol key, so the file needs no adapter package.Testing for
93a2aa46(private GCP VM, deleted after the run; tests ran in a loopback-only network namespace with synthetic media and keys):1afa12d6, Node 22.23.4 / 20.19.5 / 24.11.1,openai4.52.7 (devDependency) and 7.55.0@traceai/autosuite with the file8a1c8936+ #685fd7ffc41)Cross-lane proof on that stacked tree, with the real auto wire layer, the real
@traceai/openaispeech wrapper (#683) and the real ElevenLabs adapter (#685) sharing one manager:openai4.0.0, 4.52.7 and 7.55.0 (4.0.0 has noaudio.speech, so its speech rows return early and only transcription/translation exercise the claim; the effective speech floor is 4.52.7),@deepgram/sdk3.2.0, 4.11.3 and 5.18.0, and ElevenLabs 2.67.0 and 2.75.0. Node 20, 22 and 24; latest tier run 3 times.hideOutputs, manager PII redaction, a disabled manager and an outer owner each upload 0 and hold nothing.for awaitover the OpenAI speechresponse.bodyuploads nothing. Auto alone captures it once. The cause: provider-media claims the call but observes onlyarrayBuffer(),blob()and the publicgetReader(), a limit its docs state.Explicitly unsupported (fail closed, listed in
report().attachments.unsupported)queue_full.withAttachmentCapturePolicy).association.parentis the authoritative link. Wire-only captures held on a deferred target bind inside the emitter before the wire span ends, so their reference is also projected onto that span.IN_FLIGHTcontext) instead of a direct pass-through, so the media observer can see allowlisted routes. Span capture keeps its own gate, so no duplicate spans; with attachments OFF (the default) nothing changes.Qualification of
9da7e9f0(private GCP VM, deleted; tests in a network namespace with loopback only; synthetic media and keys)Branch tree verified identical on the VM. Control = #682 head 88a4c23 (this lane's base without the auto work).
attachments,attachment-lifecycle,attachment-adapters), Node 22.23.4 / 20.19.5 / 24.11.1openaiSDK 4.52.7 / 5.23.2 / 6.0.0 / 7.55.0 (latest)duplex(an SDK limitation, also true without auto), so that row is exercised on 5.x and later only@traceai/autojest suite, head vs control (Node 22)attachmentManageroption name, no wire JSON input, env override disables a borrowed manager, error bodies capturedPrivacy negative controls in the suite: disabled,
hideInputs,hideInputText,hideInputImages,piiRedaction,hideOutputs,hideOutputMessages,hideOutputText, serverrequire_media_redaction(T-PF-09: no base64 decode), canary filename and URL never in requests or spans, lookalike host/port, error status, non-audio MIME, raw multipart, oversized image, TOOL-kind span, policy tightened between request and consumption, hook removed, child scopes cannot loosen integration restrictions, disabled manager makes no control traffic. T-CS-01: no extraread()on the application's reader, the response body is never teed or cloned. Evidence is kept privately.Remaining
--require .../preload.cjs) and serverless fixtures are not run yet; require, existing-provider and borrowed-manager fixtures are.feat/attachments-ts-core.🤖 Generated with Claude Code