Skip to content

feat(auto): attachments in @traceai/auto (A02 ownership, A03 wire JSON/FormData, A04 consumed audio) - #687

Draft
nik13 wants to merge 23 commits into
feat/attachments-ts-corefrom
feat/attachments-ts-auto
Draft

nik13 wants to merge 23 commits into
feat/attachments-ts-corefrom
feat/attachments-ts-auto

Conversation

@nik13

@nik13 nik13 commented Nov 11, 2026 •

Copy link
Copy Markdown
Contributor

Part of TH-2379 (Linear).

What this does

Brings attachments (images, files, computer-use screenshots and speech audio) into the zero-code agent, @traceai/auto. Work packages A02, A03 and A04 of the attachments design. Attachments stay off by default: with the default disabled manager, auto's wire skip and inspection decisions are exactly the ones it made before this change.

This branch includes #682 (OpenAI/Anthropic attachment adapters, merged in, not rebased) because auto's JSON media capture calls that lane's captureOpenAIAttachments, so an SDK span and a wire observation of the same span and position share one occurrence. Once #682 lands on feat/attachments-ts-core, this diff shrinks to typescript/packages/traceai_auto/**.

A02: one manager, ownership and status

A03: allowlisted wire JSON and FormData media

  • Exact HTTPS origin + path allowlist (OpenAI speech, transcriptions, translations, images generations/edits/variations, chat completions, responses; Deepgram speak). Other hosts, ports, credentials in URLs and non-POST calls never acquire media.
  • Every Blob and buffer goes through manager.preflight at its exact size BEFORE it is read, copied or decoded (no more get data() getters).
  • JSON media (chat/responses input images and files, computer screenshots with their tool_call_id, images b64_json, responses image_generation_call) goes through the @traceai/openai adapter only: no private parser, no recursive scraping, no URL fetch.
  • Wire-only JSON is read from the wire path's own bounded request/response copies at the moment the wire span exists, so it works whether the application called json(), text() or arrayBuffer() (openai 7.x uses text()).

A04: consumed wire audio, no eager reads

  • Speech the application materialized (arrayBuffer() / blob()) is captured, bound to the actual wire span.
  • Under a native model span with an explicit integration policy, chunks the application reads (reader read() or async iterator next()) feed startAttachmentStream. Cancel, early return, pipe, tee and over-cap objects upload nothing; nothing partial is ever uploaded as complete.
  • The adapter-owner context marker (Symbol.for("traceai.auto.attachment_owner.v1"), set by the ElevenLabs adapter in feat(voice): TS LiveKit Agents recording, Pipecat recording link, ElevenLabs speech attachments #685) is honoured.

Additions in 60ef0c90 (2026-11-12): deferred targets adopted, review notes fixed

Merges feat/attachments-ts-core at 930876ea (deferred-target preflight, late/unsampled bind refusal) by merge, not rebase.

Wire-only FormData and wire-only incremental speech are now captured (previously correlation_unavailable):

  • Wire-only transcription/translation FormData audio (the openai 5.x+ SDK audio.transcriptions route) and wire-only incremental speech reads (reader or async iterator; OpenAI speech and the Deepgram SDK speak route) are admitted against a deferred target before the wire span exists. Same deny-wins admission as before; the core holds bounded bytes (at most half the owned budget) and reads or sends nothing.
  • When the emitter starts the wire span it binds the target before the span ends: policy, the environment switch and sampling are checked again there, then the capture uploads against that span (association.parent, and the reference is projected onto it).
  • Every path on which no wire span will be bound cancels the target at once and frees the bytes: unmatched route (image edits and variations have no wire span), the emitter's delivery/coverage gates, a build failure, or an invalid span identity (bind() returning false, which also covers a late bind, is treated as terminal). The bind deadline is the core maximum, 300 s, because a long transcription can take minutes; it is only the backstop (for example a hook removed mid-flight).
  • JSON media never uses a deferred target: it is still captured at bind from the wire path's own bounded copies (the feat(openai,anthropic): capture typed images, documents and computer screenshots as attachments #682 registry keys on the bound span).

Review notes:

  • report().integrations[].media reports provider_adapter for anthropic (version aware, like openai).
  • New child-process rows drive auto.init() with an ENABLED manager against allowlisted HTTPS hosts (api.openai.com speech and transcription, api.deepgram.com speak): each call uploads once, bound to a distinct exported wire span. The same calls with FI_ATTACHMENTS_ENABLED=false set after bootstrap send no attachment traffic and leave provider traffic and wire spans unchanged.
  • The env-override test now uses a manager whose environment reads process.env: it proves the kill switch denies at admission (native and wire-only, materialized, incremental and FormData) and that lifting it resumes capture on the same borrowed manager.
  • A new row runs the installed @traceai/anthropic under auto.init(): one image upload bound to the Anthropic LLM span.
  • The A02 line about the env override now states its precondition (a manager that reads process.env).

Testing for 60ef0c90 (private GCP VM, deleted after the run; tests in a loopback-only network namespace; synthetic media and keys). Control = 04e3e5d9 (this branch's previous head merged with ts-core 930876ea, no lane change).

Check Result
Attachment suites (attachments, attachment-lifecycle, attachment-adapters), Node 22.23.4 / 20.19.5 / 24.11.1 63/63 each
Deferred-path rows repeated 3x (Node 22) 18/18 each run
Real openai SDK rows x 4.52.7 / 5.23.2 / 6.0.0 / 7.55.0 3/3 each; transcription uploads once bound to the wire span on 5.x and later (4.x sends multipart as a Node stream, an SDK limitation, so that row returns early)
Full @traceai/auto suite, Node 22 head 1567 passed / 12 skipped; control 1553 passed / 12 skipped (no regressions; +14 rows)
Head tests on the control build (attribution) 15 rows fail, every one a new or flipped deferred/anthropic/env row (one is a cascade from a prior failure in the same file)
Mutations, each must fail at least one row 7/7 detected: no settle (3 rows), no deferred bind (10), bind()==false ignored (1), no deferred target (12), anthropic media flag removed (2), env disables a borrowed manager (1), no settle after emit (1)

GitHub Actions on 60ef0c90: @traceai/auto suite and the OpenAI/Anthropic attachment matrix (OpenAI 4.0.0 to latest, Anthropic 0.20.0 to latest) all pass.

Fix in 1afa12d6 (2026-11-12): a failed or aborted call drops the held capture

Review finding: the 60ef0c90 text above said a sync or async fetch failure cancels the held wire-only capture, but only the synchronous-throw path did. A rejected fetch (connection drop, AbortSignal abort after the send) emitted the error wire span with the media binder, so held FormData audio uploaded bound to that ERROR span.

  • fetch-runtime.ts complete(): when response.error is set, emit gets no media binder, and settle() cancels the deferred target at once. This now matches the synchronous-throw path. response.error covers a rejected fetch (network error or abort) and a response body that fails or is aborted while it is read.
  • Boundary (documented and tested): a call that completed with an HTTP error status (4xx/5xx) is a finished call, so its held input still binds to that wire span, as with a native instrumentor span.
  • docs/CONFIGURATION.md and the settle() comment now state this boundary exactly.
  • New rows in attachments.test.ts:
    • socket reset after the send, abort after the send, and a response body that fails mid-read. Each uploads nothing, sends no attachment control traffic, and frees the held bytes well inside the 300 s deadline. The error wire span is still emitted.
    • a completed 400 binds the held audio to its wire span.

Testing for 1afa12d6: two private GCP VMs, both deleted after the run. Tests ran in a loopback-only network namespace with synthetic media and keys. The VM tree was 60ef0c90 plus this diff (byte-identical patch).

Check Result
Attachment suites (attachments, attachment-lifecycle, attachment-adapters), Node 22.23.4 / 20.19.5 67/67 each
The 4 new rows, repeated 5x (Node 22) 4/4 each run
Full @traceai/auto suite, Node 22 (first VM, at 9e0a44a9) 1568 passed / 12 skipped. Only the 3 new failure rows failed, on a jest cross-realm toBeInstanceOf(Error) assertion. 1afa12d6 fixes the assertion, and those rows pass in the run above
Probes with real auto.init(), a process-env manager and a loopback provider reset / abort / mid-body failure: 0 uploads, held 4096 B freed right after the call, wire span status ERROR with no attachment refs. Completed 400: 1 upload bound to its wire span. Happy path unchanged: 1 upload, OK span
Mutation: bind held media on error again (the old behaviour) detected. The 3 new failure rows fail, and the reset/abort/mid-body probes flip back to 1 upload with refs present

GitHub Actions on 1afa12d6: see the checks on this PR.

Test added in 93a2aa46 (2026-11-12): one call, two layers, one upload

traceai_auto/test/attachment-owner.test.ts (test only; no source change). It pins the owner rule for one call seen by two layers: an adapter that captures claims the call with Symbol.for("traceai.auto.attachment_owner.v1") around the provider SDK call, and the wire layer then uploads nothing for that call. Before this, every owner row in the suite used a native span with arrayBuffer(); none covered wire-only incremental reads, which 60ef0c90 started capturing through a deferred target.

  • 22 rows: adapter-owned OpenAI speech and Deepgram speak read with a reader, an async iterator and arrayBuffer(), each with and without an outer capture policy. The body is read after the claim scope has ended, as an SDK caller does.
  • Also covered: FormData transcription under the claim, an outer owner with no adapter upload (0 uploads, nothing held), a concurrent owned and unowned call (the claim does not leak), and the real openai SDK.
  • The provider adapter (@traceai/provider-media, feat(providers): typed provider media capture for TypeScript Gemini, Bedrock, compatible chat and speech #683) is stubbed through the public Symbol key, so the file needs no adapter package.

Testing for 93a2aa46 (private GCP VM, deleted after the run; tests ran in a loopback-only network namespace with synthetic media and keys):

Check Result
New file on 1afa12d6, Node 22.23.4 / 20.19.5 / 24.11.1, openai 4.52.7 (devDependency) and 7.55.0 22/22 each; repeated 5x on Node 22
Full @traceai/auto suite with the file 48 suites, 1593 passed / 12 skipped
Mutation: wire layer ignores the owner mark detected, 11 rows fail (two copies)
Mutation: no wire-only deferred target detected, 4 rows fail (unowned reads upload nothing)
Same file on the stacked tree (this PR + #683 8a1c8936 + #685 fd7ffc41) 22/22

Cross-lane proof on that stacked tree, with the real auto wire layer, the real @traceai/openai speech wrapper (#683) and the real ElevenLabs adapter (#685) sharing one manager:

  • Coverage: 28/28 rows at every SDK tier: openai 4.0.0, 4.52.7 and 7.55.0 (4.0.0 has no audio.speech, so its speech rows return early and only transcription/translation exercise the claim; the effective speech floor is 4.52.7), @deepgram/sdk 3.2.0, 4.11.3 and 5.18.0, and ElevenLabs 2.67.0 and 2.75.0. Node 20, 22 and 24; latest tier run 3 times.
  • No call uploads twice. That includes wire-only incremental OpenAI speech and Deepgram speak with provider-media and ElevenLabs active, and a concurrent ElevenLabs + OpenAI + Deepgram run that produced exactly three uploads, each bound to its own owner span.
  • Privacy: manager hideOutputs, manager PII redaction, a disabled manager and an outer owner each upload 0 and hold nothing.
  • Mutations were all detected: no OpenAI claim, provider-media ignores the owner, auto ignores the owner, no deferred target.
  • Finding (an ask on feat(providers): typed provider media capture for TypeScript Gemini, Bedrock, compatible chat and speech #683, not a double upload): with provider-media active, for await over the OpenAI speech response.body uploads nothing. Auto alone captures it once. The cause: provider-media claims the call but observes only arrayBuffer(), blob() and the public getReader(), a limit its docs state.

Explicitly unsupported (fail closed, listed in report().attachments.unsupported)

  • Wire-only image edits and variations: these routes have no wire span, so a held file is dropped as soon as the call completes.
  • Wire-only FormData or incremental speech above the pre-bind budget (half the owned budget): queue_full.
  • Native model spans without an explicit integration capture policy (withAttachmentCapturePolicy).
  • Raw multipart and streamed request bodies; compressed responses consumed before the span is emitted; RemoteUrl / ProviderFile; WebSocket / RTC; new recordings.
  • association.parent is the authoritative link. Wire-only captures held on a deferred target bind inside the emitter before the wire span ends, so their reference is also projected onto that span.
  • No product playback claim: Q03 is not run here.
  • Known behavioural difference when attachments are ON: calls under native model coverage now go through the fetch Proxy path (inspection, IN_FLIGHT context) instead of a direct pass-through, so the media observer can see allowlisted routes. Span capture keeps its own gate, so no duplicate spans; with attachments OFF (the default) nothing changes.

Qualification of 9da7e9f0 (private GCP VM, deleted; tests in a network namespace with loopback only; synthetic media and keys)

Branch tree verified identical on the VM. Control = #682 head 88a4c23 (this lane's base without the auto work).

Check Result
Attachment suites (attachments, attachment-lifecycle, attachment-adapters), Node 22.23.4 / 20.19.5 / 24.11.1 49/49 each
Same suites x real openai SDK 4.52.7 / 5.23.2 / 6.0.0 / 7.55.0 (latest) 49/49 each. On 4.x the SDK transcription row returns early: openai 4.x sends multipart as a Node stream, which a WHATWG fetch rejects without duplex (an SDK limitation, also true without auto), so that row is exercised on 5.x and later only
Full @traceai/auto jest suite, head vs control (Node 22) head: only 2 test-harness rows failed in round 2 (both fixed, then 49/49 above); docs suite green; control 1504/1504 with 12 skipped
Head attachment tests against the control build all fail (the code they test is absent): attribution
Mutation negative controls (each must fail >= 1 row) 10/10 detected: native capture without policy, ignored owner marker, no output dedup, cancel commits, early return commits, loose host match, old attachmentManager option name, no wire JSON input, env override disables a borrowed manager, error bodies captured

Privacy negative controls in the suite: disabled, hideInputs, hideInputText, hideInputImages, piiRedaction, hideOutputs, hideOutputMessages, hideOutputText, server require_media_redaction (T-PF-09: no base64 decode), canary filename and URL never in requests or spans, lookalike host/port, error status, non-audio MIME, raw multipart, oversized image, TOOL-kind span, policy tightened between request and consumption, hook removed, child scopes cannot loosen integration restrictions, disabled manager makes no control traffic. T-CS-01: no extra read() on the application's reader, the response body is never teed or cloned. Evidence is kept privately.

Remaining

🤖 Generated with Claude Code

nik13 and others added 16 commits November 10, 2026 19:59
… attachment manager

OpenAI chat/responses input images, files and computer_call_output
screenshots, Responses image_generation_call and Images generate b64
outputs; Anthropic Messages image/document blocks and nested tool_result
media with computer-tool screenshot classification. Decode is lazy behind
manager admission, inline only and capped at 256 KiB per part. Telemetry
payloads are projected so media bytes and filenames never reach OTLP.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…telemetry otherwise

Capture now runs before the telemetry projection and returns the admitted
positions. Only those parts are replaced by their fi-attachment occurrence
URI; denied, disabled or absent capture leaves the span payload exactly as
before. A second observer of the same span position reuses the first
occurrence. Streamed Responses output media and the Images edit and
variation endpoints are explicitly unsupported. Adds a network-isolated
CI workflow against real SDK floor and latest releases.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…dently of the manager

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…r SDK typings

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…d data URIs

The cloud mutation run showed the OpenAI cap was not exercised: the oversized
case was not a data URI, so it failed the prefix check first.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…nai-anthropic

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…Responses output

Adopt the core preflight API (sdk-streaming-design section 1) in place of
lazy `get data()` getters:

- Each inline part is preflighted (deny-wins, budget reserved) before its
  value is read or decoded. Denied parts are never read.
- An admitted ticket that is not used (malformed base64, MIME outside the
  allowlist, decode failure) is released at once, not left to the TTL sweep.
- OpenAI data URIs naming a sibling MIME of the same modality are released
  and re-admitted under the exact MIME before decoding.
- When the server negotiates an upload limit below the 256 KiB inline cap,
  preflight returns too_large only after every privacy step passed, so the
  part is re-preflighted with its exact decoded length. Small media keeps
  capturing as before.

Streamed Responses: the generated image in the terminal response.completed
event that the application consumed from its own iterator is now captured
and projected out of the span output, before the span ends. Partial
(preview) images and streamed images.generate stay unsupported and are
named in the capability descriptor.

Test fixtures mint one-hour sessions (core renewal window), assert no
FI_ATTACHMENTS_* degradation warnings and no leaked tickets or bytes.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ct size

A cap-sized (256 KiB) preflight can return queue_full when the owned-byte
budget is nearly used, where the previous attach() reserved only the actual
length and admitted small media. queue_full, like too_large, is evaluated
after every privacy, correlation and sampling step, so the part is sized
exactly and preflighted once more. New test row for both providers.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Recovered from refs/recovery/att-b-auto-1111 (base 7145f78), applied on
feat/attachments-ts-core f0a56b3. Not yet migrated to the preflight API.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ream capture

A02: pass the shared manager to instrumentors as `attachments` (the option every
adapter lane uses), keep media status apart from span status, and stop disabling a
borrowed manager on an env override (updatePolicy is tighten-only; the core applies
the override at admission). With attachments OFF, wire skip/inspection is unchanged.

A03: every FormData Blob and speech buffer is preflighted at its exact size before it
is read; JSON media (chat/responses inputs, image outputs, computer screenshots) goes
through the @traceai/openai adapter so SDK and wire observations of one span share an
occurrence. Wire-only JSON input uses the wire path's own bounded request copy.

A04: chunks the application reads (reader read() or async iterator next()) feed
startAttachmentStream under a native model span with an explicit capture policy.
Cancel, early return, pipe, tee and clone abort; nothing partial is uploaded.
Wire-only incremental reads and wire-only FormData stay unsupported until the core
can preflight a late-bound target.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… fix test doubles

The latest OpenAI SDK (7.x) consumes JSON with text(), so wire-only image output is
now read from the wire path's own bounded response copy at bind, whatever method the
application used. Test doubles read Blob request bodies; T-CS-01 ignores undici's own
request-body tee and reads.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…s exports map

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
nik13 and others added 7 commits November 11, 2026 17:07
… deferred targets

Merges feat/attachments-ts-core 930876e and adopts its deferred-target
preflight in the wire media observer:

- Wire-only FormData audio (OpenAI transcriptions/translations, the
  openai 5.x+ SDK route) and wire-only incremental speech reads (reader or
  async iterator; OpenAI speech, Deepgram speak) are admitted against a
  deferred target before the wire span exists. The core holds bounded
  bytes, reads and sends nothing, and re-checks policy, environment and
  sampling when the wire span binds.
- Every path on which no wire span will be bound (unmatched route, sync
  or async failure, emitter delivery gate, build failure, invalid span
  identity) cancels the target at once; the 300 s bind deadline is only
  the backstop. A late bind returning false is treated as terminal.
- JSON media is unchanged: captured at bind from the wire path's own
  bounded copies, never through a deferred target.
- report().integrations[].media reports provider_adapter for anthropic
  (version aware, like openai).

Tests: wire-only FormData and reader/iterator rows flipped from "not
captured" to one upload bound to the exported wire span; no-span,
unsampled, image-edit (no wire span) and policy-tightened-while-held rows;
deferred path inside the privacy matrix; the env-override test now builds
a process-env manager and proves the kill switch at admission; new
child-process rows drive auto.init() with an enabled manager against
allowlisted HTTPS hosts (and with the kill switch flipped after
bootstrap), and the installed @traceai/anthropic instrumentor end to end.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… at once

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…wire-only materialized refs now project onto the wire span

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ture

A fetch that rejected (connection drop, AbortSignal abort) or whose response
body failed while read still emits an error wire span, but the deferred
target held for wire-only FormData audio or incremental speech is no longer
bound to it: emit gets no media binder, and settle() cancels the target at
once, as on the synchronous-throw path. A call that completed with an HTTP
error status is a finished call and still binds.

Regression rows: socket reset, abort after send and a response body that
fails mid-read each upload nothing and free the held bytes; a completed 400
binds the held audio to its wire span. CONFIGURATION.md and the settle()
comment now state the exact boundary.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…rors come from Node's realm)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
One call seen by two layers: an adapter that captures claims the call with
Symbol.for("traceai.auto.attachment_owner.v1") around the provider SDK call,
and the wire layer then uploads nothing for that call. This adds the rows
the suite did not have: wire-only incremental reads (reader and async
iterator, now captured through a deferred target) and FormData under the
claim, for the OpenAI speech and Deepgram speak routes, with and without an
outer capture policy, an outer owner with no adapter upload, a concurrent
owned and unowned call (the claim does not leak), and the real OpenAI SDK.

The provider-media claim is stubbed through the public Symbol key, so the
file needs no adapter package.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant