Skip to content

feat(providers): typed provider media capture for TypeScript Gemini, Bedrock, compatible chat and speech - #683

Draft
nik13 wants to merge 21 commits into
feat/attachments-ts-corefrom
feat/attachments-ts-providers
Draft

nik13 wants to merge 21 commits into
feat/attachments-ts-corefrom
feat/attachments-ts-providers

Conversation

@nik13

@nik13 nik13 commented Nov 11, 2026 •

Copy link
Copy Markdown
Contributor

Part of TH-2379 (Linear).

What this adds

A shared @traceai/provider-media helper and thin bindings, so TypeScript provider packages can hand typed media (images, PDFs, audio) to the single attachment manager from the TypeScript attachment core (#677). Attachments stay off unless the application passes an enabled manager. Every privacy decision is made by the manager before any source bytes are read.

Stacked on feat/attachments-ts-core (#677).

Supported operations (exactly these; see PROVIDER_MEDIA_CAPABILITIES)

Package Operation Media
@traceai/google-genai-sdk models.generateContent inline Part input and generated output
@traceai/google-genai-sdk models.generateContentStream inline Part input only (stream output is not captured)
@traceai/google-genai GenerativeModel.generateContent inline Part input and output
@traceai/fi-instrumentation-vertexai GenerativeModel.generateContent inline Part input and output
@traceai/bedrock ConverseCommand on allowlisted Claude and Nova model ids image/PDF byte blocks, including typed toolResult blocks (tool screenshots)
@traceai/groq chat, transcriptions, translations, speech data-URI images; File/Blob audio input; speech bytes the application itself reads (whole body, or its own reader to the end)
@traceai/mistral chat.complete data-URI imageUrl chunks
@traceai/portkey, @traceai/vllm chat.completions.create data-URI images, typed input_audio
@traceai/openai audio transcriptions, translations, speech audio only. The OpenAI chat hook belongs to the OpenAI adapter work and is not touched here

Explicitly unsupported (fail closed, never captured)

  • Remote, signed or gs:///s3:// URLs, provider file ids, filesystem paths. Nothing is fetched or read.

  • Gemini generateContentStream output, Bedrock ConverseStream, and completed-recording imports. Speech bodies consumed through tee(), pipeTo(), pipeThrough() or clone(), and chunk-wise reads (for await, pipe, on('data')) of Node-stream bodies, such as the node-fetch 2 body that openai 4.x uses on its default Node transport. The auto wire layer uploads nothing for these either (it captures through fetch only, so on the node-fetch transport it uploads nothing for any read style).

  • Async operations (submitted video/image jobs), Bedrock InvokeModel bodies, unknown Bedrock model families, ConverseStream.

  • Unknown or text-only hosts. Mistral 0.x clients, which are text-only.

  • The purposes and roles used (model_input, model_output, tool_result; role tool) match the frozen contract's Purpose and Reference.role enums.

Privacy and lifetime

  • Every part is preflighted (manager.preflight) before its source is read, sized or decoded, then attached with the ticket (ticket.attach). Disabled, hidden-input, hidden-image, PII-redaction and unsampled cases never touch the source (tests use throwing getters to prove this).
  • Wrapper-local traceConfig restrictions are intersected with the manager policy; they can only deny.
  • Speech output is observed only when the application calls arrayBuffer()/blob(), reads a reader it got from response.body.getReader() to the end, or iterates a WHATWG response.body to the end (for await, values()). On a node-fetch (Node-stream) body only a whole-body arrayBuffer() read is observed; blob() resolves node-fetch's own Blob class and is not. Unread, cancelled, partially read, released or broken-out-of bodies upload nothing, and the wrapper never clones, tees or reads the response itself.
  • OpenAI audio calls get spans only while an attachment manager is active, so the default path is unchanged.
  • Per-span, per-position deduplication, so the same media observed twice on one span is uploaded once. A 64-part budget per call.
  • No bytes, base64 or filenames reach span attributes or upload metadata.

Package location and release entry (fix pass)

  • The helper lives at typescript/packages/provider-media, beside fi-core, not under traceai_*. The @traceai/auto registry test counts every typescript/packages/traceai_* directory as an instrumentation, so the old traceai_provider_media location failed it (52 directories vs 51 registered integrations).
  • release-please-config.json and .release-please-manifest.json have an entry for it (release-type: node, component npm-traceai-provider-media, 0.1.0). Without it, release_plan.py --check and the release unit tests failed, and npm_release.py would refuse every release of the 9 dependent packages. Adding the entry publishes nothing.
  • Regression test typescript/packages/provider-media/test/layout.test.cjs guards both: the directory must stay outside traceai_*, dependent tsconfig references must point at it, and the release-please config and manifest entry must match package.json. It runs in the package test script and as the first step of the provider-media workflow, so it runs on this stacked PR even though Release and the @traceai/auto workflow only run against dev/main.

CI

  • New TypeScript provider media attachments workflow: helper unit tests, plus real provider SDKs at their floor and pinned latest versions against loopback servers inside a network namespace with no route out. A packed-helper consumer check, CJS and ESM.
  • Workflows that install packed @traceai/openai, @traceai/google-genai or @traceai/google-genai-sdk now also stage the helper tarball, and dependent workflows trigger on helper changes.

Not in this PR

  • Publishing @traceai/provider-media. Its release-please entry is in place (see the fix pass below); it has to be published before any dependent provider release, and that stays a release decision.
  • Checking that OpenAI-compatible wrappers that delegate to OpenAIInstrumentation (Fireworks and similar) inherit chat capture. That waits for the OpenAI chat adapter to land.
  • Together/xAI native classes and Ollama/OpenRouter native media.

Qualification

Run on a temporary private cloud VM (Node 24.11.0 and 20.19.5). The package regression suites ran at f896ae45; later commits changed only helper tests, the new workflow and the Groq input_audio dialect gate, which CI re-qualifies. Synthetic data only. Provider SDK tests ran in a network namespace that had loopback only.

  • Frozen install and build of the helper plus all 9 bound packages: passed.
  • Helper unit tests: 27/27 on Node 24 and on Node 20. They cover privacy negative controls with throwing source getters (disabled, hideInputs, hideInputMessages, hideInputImages, hideInputText, piiRedaction, wrapper-local hide/PII, unsampled), speech arrayBuffer/blob/unread/cancel/stream, late policy tightening, malformed and oversized base64, the data-URI part budget, Bedrock tool-result blocks and capability descriptors.
  • Real SDKs against loopback, 20/20 per tier on both Node versions (80 runs in total):
SDK floor latest (pinned)
openai 4.0.0 7.55.0
groq-sdk 0.3.0 1.6.0
@google/genai 1.33.0 3.1.0
@google/generative-ai 0.1.0 0.24.1
@google-cloud/vertexai 1.0.0 1.12.0
@aws-sdk/client-bedrock-runtime 3.693.0 3.1166.0
@mistralai/mistralai 1.1.0 (first release with typed image chunks) 3.5.0
portkey-ai 1.0.0 3.1.0
  • Existing package suites, unchanged behaviour: @traceai/openai 304 passed / 2 skipped, @traceai/groq 141 passed / 5 skipped, @traceai/mistral 31 jest + 79 native, @traceai/portkey 322, @traceai/vllm 22, @traceai/google-genai 30, @traceai/fi-instrumentation-vertexai 17, @traceai/bedrock 8, @traceai/google-genai-sdk suite passed.
  • Packed consumers: helper CJS and ESM; @traceai/openai in an npm consumer and in a non-hoisted pnpm consumer (strictImports); the standalone @traceai/google-genai-sdk build used by its compat workflow.
  • GitHub CI on head f6b38013: all 12 checks green. The new provider-media workflow passed on Node 20 and 24: 28/28 helper tests, and 20/20 real-SDK tests for each of the floor and latest tiers. The 9 Google GenAI SDK packed-consumer jobs also passed.
  • Not yet run in CI: the dependent workflows filtered to dev/main (OpenAI contracts, OpenAI gateway, Fireworks, legacy Gemini, Groq, Mistral, Portkey, vLLM). They run when this stack targets dev. Of their packed-consumer steps, the OpenAI npm and non-hoisted pnpm consumers and the Google GenAI SDK standalone build were simulated on the VM. The legacy Gemini, Fireworks and gateway consumer steps were not simulated.

Fix-pass qualification (head 08056560)

Run on temporary private cloud VMs, Node 24.11.0, pnpm 10.30.1, from pristine trees: on 5b20ea51, and on a local merge of 08056560 with the current feat/attachments-ts-core (149fef0a), which is what PR CI tests. Every gate below passed on both. All three VMs were deleted and verified gone.

  • Base drift: on the merge with the newer core, the earlier helper test that read uploads[63].part_index failed 3 out of 3 runs (62 !== 63). The manager runs two upload workers, and the new core queue changes their completion order. CI on 5b20ea51 hit the same failure. The test now checks that exactly parts 0..63 were uploaded, which is the budget it is meant to prove.
  • Negative control on the reviewed head f6b38013: the new layout test fails (exit 1) and release_plan.py --check fails (exit 1).
  • Release gates: python3 -m unittest discover -s .github/release 87 tests OK; release_plan.py --check "156 packages, versions and coverage consistent"; npm_release.dependency_closure orders @traceai/provider-media before each of its 9 dependents.
  • Workspace: pnpm install --frozen-lockfile, then pnpm install --lockfile-only leaves pnpm-lock.yaml unchanged; pnpm run build:ordered passes.
  • @traceai/auto: the registry test passes, and the whole test/config.test.ts passes 12/12.
  • Helper: layout 2/2, package test script 30/30, helper unit tests 28/28 in a loopback-only network namespace, real SDKs 20/20 on the floor tier and 20/20 on latest. Packed helper installs into a clean npm consumer and loads as CJS and ESM. No traceai_provider_media reference remains.

A pre-existing issue found along the way, outside this PR's scope: @traceai/fi-instrumentation-vertexai keeps process-wide patch state. A second copy of @google-cloud/vertexai in the same process is never patched, so the SDK tests run each tier in its own process.

Re-proof on head 7bddcd16 (2026-11-11)

Head 7bddcd16 is 08056560 merged with feat/attachments-ts-core f054a68e; it adds no provider changes. Run again on a temporary private cloud VM (Ubuntu 24.04, pnpm 10.30.1) from pristine clones, against the reviewed head f6b38013 as the negative control. The VM was deleted and verified gone.

Gate 7bddcd16 control f6b38013
python3 -m unittest discover -s .github/release -p 'test_*.py' 87 OK 1 failure (test_real_repo_has_complete_manifest_and_runtime_markers)
release_plan.py --check "156 packages, versions and coverage consistent" exit 1, missing release-please entry
test_release_strategies.cjs (release-please@17.6.0) and bootstrap_audit.py --check pass pass
No traceai_provider_media reference left (git grep) pass fails (workflows still use it)
pnpm install --frozen-lockfile, then --lockfile-only leaves the lockfile unchanged pass lockfile not canonical
pnpm run build:ordered pass pass
@traceai/auto registry test / test/config.test.ts pass / 12/12 fails, "Expected length: 52, Received length: 51"
Full @traceai/auto suite, Node 22.23.4 44/44 suites, 1504 passed, 12 skipped 1 failed (the registry test), 1503 passed
@traceai/provider-media package test script pass not run (helper path moved)
  • Mutation checks on 7bddcd16: removing the release-please config entry fails the layout test and release_plan.py --check; pointing traceai_groq/tsconfig.json back at ../traceai_provider_media fails the layout test; restoring them passes it again (2/2).
  • On Node 22.11.0, 7 test/bundle.test.ts cases fail on both commits, because that Node prints an import-in-the-middle "hook has already been initialized" warning to stderr and the tests require empty stderr. This comes from the Node version, not from this PR. On Node 22.23.4, the version CI's setup-node resolves 22 to, the full suite passes, as above.
  • GitHub CI on 7bddcd16: every check is green, including a dispatched @traceai/auto suite run (44 suites, 1504 passed, Node 22.23.4). The Release workflow was not dispatched, because even its dry run plans releases for the whole repo from a feature branch. The cloud VM covered its plan-job gates (above), and Release runs for real once this stack targets dev.

Resync with the TypeScript attachment core f0a56b37 (head 9accc54e)

  • a286cd7d merges feat/attachments-ts-core f0a56b37 (clean). On that merge, with no lane change, the old helper tests pass 30/30 and the real SDKs 20/20 per tier, so capture never stopped.
  • db010c66 moves the helper from lazy get data() getters through manager.attach to manager.preflight + ticket.attach, as the SDK design requires (section 1):
    • A gate preflight runs before any getter, URL string, byte array or base64 is read. Its size is the 1 MiB base64 cap, or the known Blob/ArrayBuffer length.
    • After admission, the part is sized from its shape and decoded, and the reservation shrinks to the exact length.
    • Some parts are preflighted once more at their exact type and length: a gate denied only for size (too_large/queue_full come after every privacy, identity and sampling check), a part larger than the gate, or a data URI of another image type. There are never more than two preflights per part.
    • Unused tickets are released at once. A manager without preflight captures nothing. Bedrock bytes have no adapter cap beyond the manager's.
    • Speech bodies are preflighted when the application reads them, never when the hook is installed.
  • aca3ecca adopts the consumed-stream API (design 6.4) for speech output. When the application calls response.body.getReader(), each chunk its own read() resolves goes to startAttachmentStream (4 MiB reserved). The stream finishes when the application reads to the end. It aborts on releaseLock, cancel, a read error, a tightened policy or the cap. One span uploads once, whichever path read it.
  • 9accc54e observes only the first reader on a body. releaseLock() does not end a stream, so on aca3ecca a second reader uploaded only the tail of the body as if it were whole. A second reader now uploads nothing (row S-08). Descriptor change: audio.speech.create (openai, groq) adds source consumed_reader.
  • New tests: test/preflight.test.cjs (P-01..P-09) and test/speech-stream.test.cjs (S-01..S-08). The provider-media workflow runs both. The media row for a reader read to its end now expects one upload.

Qualified on a temporary private cloud VM (deleted and verified gone). It used the CI install and build commands, with tests in a loopback-only network namespace. The VM rebuilt all three trees from the public refs plus patches, and each tree matched the local commit.

Gate Result
Frozen install and build of the helper plus the 9 bound packages, on control, preflight-only and head pass; lockfile unchanged
Head helper tests (layout, media, preflight, speech-stream), Node 20.19.5 / 24.11.1 51/51 / 51/51; preflight + stream files 21/21 in 3 repeats
Head real SDKs against loopback, floor and pinned latest, Node 20 / 24 20/20 per tier on each
Control (merged core, old helper): its own helper tests / real SDKs 30/30 / 20/20 per tier
Head tests on the control build 41/49: fails P-01, P-02, P-03, P-04, P-08, S-01, S-02 (abandon) and the media stream row
Head tests on the preflight-only build 46/49: fails only S-01, S-02 (abandon) and the media stream row
FI_ATTACHMENTS_* warnings in any run 0
Packed helper in a clean consumer, CJS and ESM pass
Mutations: read before preflight / no ticket release / no size retry / finish on releaseLock / drop chunks detected: 9 / 1 / 2 / 1 / 2 rows fail; head restored 51/51
Round 2, head 9accc54e (second VM, deleted): helper tests Node 20 / 24; real SDKs Node 24 52/52 / 52/52 (speech-stream 13/13 in 3 repeats); 20/20 per tier
Round 2: head speech-stream tests on the aca3ecca build 12/13; only S-08 fails

Still open (resolved in 32e16cb1, see below): when the server's max_upload_bytes is below 4 MiB, the reader path is denied too_large and captures nothing. The whole-body paths are unaffected, because they are preflighted at their exact size.

Shared media-owner marker and the clamped stream reservation (head 8a1c8936)

The auto module's wire layer and the ElevenLabs instrumentation already mark a call whose media they capture with the OpenTelemetry context key Symbol.for("traceai.auto.attachment_owner.v1"). Before this change the provider-media speech path neither set nor read it, so auto plus @traceai/openai speech could upload the same audio twice.

  • 68c3eedf / 5044417a merge feat/attachments-ts-core (01b3327a, then 930876ea). The changes are in fi-core only; this helper uses no deferred targets, so bind() returning false for a late bind does not affect it.
  • 32e16cb1, @traceai/provider-media:
    • Exports ATTACHMENT_OWNER_KEY (the same Symbol.for key; createContextKey is Symbol.for, so no package imports another), attachmentOwnedElsewhere(ctx) and ProviderMediaCapture#claim(ctx).
    • Honour: a capture built with a parentContext that already carries the marker never reaches the manager (no preflight, no stream, no upload); the application still gets every byte.
    • Set: openai audio.speech/transcriptions/translations and groq audio calls run the SDK call in a claimed context, so wire-level capture skips them and only the provider adapter uploads. A locally hidden call is still claimed, so the integration's hide holds for every layer. Calls on unqualified hosts, calls with attachments off and Groq chat are not claimed.
    • Scope: speech/audio only. Gemini, Vertex, Bedrock and compatible chat do not use the marker. The marker is only for a wire observer and an adapter seeing the same call; each observing span otherwise owns its own occurrence.
  • min(4 MiB, server max_upload_bytes): this needs no helper code. The core (01b3327a) clamps the consumed-stream reservation to the negotiated single-upload cap. The reader path keeps asking for 4 MiB, now admits on servers below 4 MiB, and a body over the effective cap is discarded whole. Rows S-09/S-10 cover this.
  • 5012a6f9 / 8a1c8936: test-only.
    • On a non-default base URL, the latest OpenAI SDK also fetches a data: URL through the custom fetch. The unqualified-host row now asserts the provider call by route, and asserts that no request carries the marker.
    • The speech and owner SDK rows expect the OpenAI chat adapter's single chat-image capture when that adapter is in the same build (stacked lanes). Standalone, the expectation stays 0, and a second copy still fails.
  • New tests: test/owner.test.cjs (O-01..O-05), S-09/S-10, and real OpenAI/Groq SDK rows. The SDK rows read the marker from the SDK's own fetch context, and an owned call uploads nothing.

Qualified on a temporary private cloud VM, now deleted and verified gone. It used the CI install and build commands, with tests in a loopback-only network namespace. Each VM tree matched the local commit tree.

Gate Result
Frozen install and build, head and control (9accc54e + core 930876ea, no lane change) pass; lockfile unchanged
Head helper tests (layout, media, preflight, speech-stream, owner), Node 20.19.5 / 22.23.4 / 24.11.1 60/60 each; owner + stream 21/21 in 3 repeats
Head real SDKs, floor and pinned latest, Node 20 / 24 24/24 per tier on each (5012a6f9 and 8a1c8936)
Packed helper in a clean consumer, CJS and ESM (exports the owner key) pass
Head tests on the control build owner + speech-stream 17/21: the 4 failures are O-01, O-02, O-03, O-05. S-09/S-10 pass on control too, because the clamp is in the merged core (the core-clamp mutation below shows S-09 depends on it). SDK latest 22/24 (OpenAI and Groq owner rows fail)
Mutation: drop the OpenAI claim / drop the Groq claim / ignore an existing owner / remove the core clamp detected: OpenAI owner row / Groq owner row / O-02, O-03 + both SDK owner rows / S-09; head restored 60/60
Cross-lane, auto + provider-media on one manager: this head merged with feat/attachments-ts-auto (9da7e9f0); the only conflict, in traceai_openai/src/instrumentation.ts, is resolved by keeping both sides. Real auto patchFetch + wire observer and the real @traceai/openai audio wrapper; Node 22.23.4 11/11 on openai floor and latest (latest 3 repeats). On floor (openai 4.0.0, no audio.speech) the speech rows and the first positive control return early, so floor exercises only transcription/translation, deny-wins, outer-owner and attachments-off. On latest, Speech arrayBuffer, speech reader, and transcription/translation File input each upload exactly once, with or without an explicit auto capture policy, bound to the OpenAI audio span. Positive controls show auto alone captures. An outer ElevenLabs-style owner, a local hide, and attachments off upload nothing
Cross-lane mutation: drop the OpenAI claim 4 rows fail on latest. Speech arrayBuffer, reader and transcription under an explicit auto capture policy upload two copies, and a local hide no longer stops auto's copy. Floor passes: openai 4.0.0 has no speech, and its multipart body is not a native FormData, so auto cannot see it
Cross-lane mutation: ignore an existing owner the outer-owner row fails (the provider adapter uploads under an ElevenLabs-style marker)
Merge-tree regressions: auto attachment suites / provider-media / OpenAI-Anthropic adapter suite (openai 7.55.0) 49/49 / 60/60 + SDK 24/24 per tier / 50/50

The cross-lane test needs auto's src/wire/fetch, so it cannot live on this branch. It goes into the auto PR (or the integration branch) once this branch is merged there. GitHub CI on 8a1c8936: 29/29 checks pass, including Provider media on Node 20 and 24 (owner, S-09/S-10 and SDK owner rows).

Async iteration of the speech body (018f3582)

  • Gap: with this PR the OpenAI speech call is claimed, so the auto wire layer skips it. for await and values() lock the body with an internal reader, not the public getReader(), so an iterated speech body uploaded 0 copies. Auto alone uploads 1. This was found by the auto cross-lane proof.
  • Fix: values() and [Symbol.asyncIterator] on the body instance now go through the same consumed-stream path as a reader:
    • each chunk the application's own next() resolves is forwarded after its promise fulfils, and done finishes the stream;
    • return() (a break), a rejected next(), a non-byte chunk, the cap or a tightened policy abort it, so nothing partial uploads;
    • only the first reader or iterator on a body is observed, so values({ preventCancel: true }) followed by a reader uploads nothing;
    • a locked body still throws the host error unchanged, and the adapter never pulls a chunk itself.
  • New rows S-11..S-17: for await and values() to the end (one upload, and pulls() == 0 after the iterator is obtained), break, preventCancel then a reader, over the cap, an error mid-iteration, the four privacy modes, and a locked body.
  • GitHub CI on 018f3582: 29/29. Provider media on Node 20 and 24 ran all 69 helper rows, including S-11..S-17, plus SDK 24/24 per tier.
  • Done: the VM cross-lane re-run; see "Cross-lane VM proof on 34809908" below.

Node-stream bodies stay unobserved for chunk-wise reads (afb4e5ae, 34809908)

  • Bug in 018f3582: the iteration hooks matched any body with [Symbol.asyncIterator]. On openai 4.x's default Node transport the body is node-fetch 2's PassThrough, which has one, so for await over a speech body uploaded 1 copy, contrary to the documented contract and unlike the auto wire layer (0).
  • Fix: reader and iterator hooks are installed only on WHATWG bodies (getReader present). Whole-body reads are unchanged.
  • On the node-fetch transport the two layers are: chunk-wise reads, auto 0 and provider-media 0; arrayBuffer(), auto 0 (it never sees the call) and provider-media 1; blob(), auto 0 and provider-media 0 (node-fetch's own Blob class is not the global Blob).
  • New rows: S-18 (a Node PassThrough body gets no hook; iterating to the end or breaking uploads nothing). The real-SDK suite has a new openai4 tier (4.52.7 / 4.104.0) driving the SDK's own node-fetch against loopback. It asserts the body is a Node Readable; for await to the end and break upload 0; arrayBuffer() uploads 1 when enabled or hideInputs and 0 when disabled or piiRedaction; blob() uploads 0 in every mode.
  • Done: the VM cross-lane run with the node-fetch block and the gate-removal mutation; see below.

Cross-lane VM proof on 34809908

Run on a temporary private cloud VM (Ubuntu 24.04, pnpm 10.30.1, lockfile unchanged), deleted and verified gone. Tests ran in a loopback-only network namespace.

  • Tree X: auto 93a2aa46 merged with this head 34809908 and voice fd7ffc41. The only conflict was OpenAI instrumentation.ts, resolved as the auto side plus the patchSpeech/unpatchSpeech lines. Control C: the same merge with the old 8a1c8936.
  • Cross-lane test (73 rows), real auto wire layer + @traceai/openai + provider-media + ElevenLabs on one manager: 73/73 on X with openai 4.0.0 / 4.52.7 / 7.55.0 on Node 22 (4.0.0 has no audio.speech, so the effective speech floor is 4.52.7), 4.52.7 and 7.55.0 on Node 20 and 24, and 7.55.0 three times. It covers every speech read style (arrayBuffer, blob, reader, for await, values, values with preventCancel): exactly one upload per call, bound to the OpenAI audio span with provider-media active and to the wire span with auto alone. break uploads 0, and privacy modes and a local hide upload 0. The node-fetch transport block (openai 4.52.7 and 4.104.0) gives for await/break 0, blob() 0 and arrayBuffer() 1 with provider-media.
  • Other suites on X: provider-media helper 71/71 on Node 20/22/24; real SDKs 28/28 at floor and latest; auto attachment-owner 22/22.
  • Control C: exactly 8 rows fail, all with 0 uploads: the iterator row and for await / values / values with preventCancel with provider-media active, each with and without a policy. Nothing else fails.
  • Mutations (each caught, then restored to 73/73 and 71/71):
    • no iterator observe: 8 cross-lane rows + S-11/S-13;
    • return() not aborting: the break row + S-12/S-13;
    • no shared first-consumer flag: S-08/S-13;
    • WHATWG gate removed: both node-fetch for await rows + both S-18 rows;
    • plus the four earlier cross-lane mutations (no OpenAI claim 12, provider ignores owner 1, auto ignores owner 15, no wire deferred target 12).

🤖 Generated with Claude Code

nik13 and others added 9 commits November 10, 2026 20:00
Shared @traceai/provider-media helper plus thin bindings for Gemini (three
wrappers), Bedrock Converse, compatible chat messages and OpenAI/Groq speech.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- Drop the OpenAI chat capture line: the OpenAI chat adapter owns that hook.
- OpenAI audio spans exist only while an attachment manager is active, and
  speech capture is limited to qualified audio hosts.
- One part budget per data URI, Bedrock typed tool-result image/PDF blocks,
  explicit per-package capability descriptors.
- New provider-media workflow (unit + real SDKs against loopback) and stage
  the helper tarball in workflows that install packed packages.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- Mistral typed image chunks start at 1.1.0 (1.0.0 is the legacy client).
- Older OpenAI shims pass node-fetch options; the loopback redirect drops them.
- Newer Bedrock clients default to HTTP/2; the fixture uses an HTTP/1.1 handler.
- Run each SDK tier in its own process (Vertex keeps process-wide patch state).
- Pin the latest qualified SDK releases for reproducible CI.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ch input

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…vertise it

Groq chat is image-only, so its descriptor and behaviour now agree.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…entry

The shared helper lived at typescript/packages/traceai_provider_media, so the
@traceai/auto registry test counted it as an instrumentation (52 vs 51). Move it
to typescript/packages/provider-media, beside fi-core, and update the tsconfig
references, workflow paths and lockfile importer.

The package is publishable, so release_plan.py inventories it; add its
release-please config and manifest entry (0.1.0) so the Release gates pass and
npm_release.py can resolve it as a dependency. Nothing is published by this.

Add test/layout.test.cjs as the regression test for both, run from the
package test script and the provider-media workflow.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…-media

pnpm install --lockfile-only reorders the @traceai/provider-media dependency
entries alphabetically; commit that output so the lockfile check stays clean.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…rder

The attachment manager runs two upload workers, and the newer TS core queue on
feat/attachments-ts-core finishes them in a different order, so uploads[63] can
be part 62. Assert that exactly parts 0..63 were uploaded instead.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…viders

Brings in the ts-core renewal fix and qualification fixes so the providers
lane is tested against its current base.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
nik13 and others added 4 commits November 11, 2026 04:43
…viders

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Migrate @traceai/provider-media from lazy `get data()` getters through
manager.attach to the SDK design's preflight + ticket.attach (section 1,
T-PF-09):

- A gate preflight (adapter cap, or the known Blob/ArrayBuffer length) runs
  before any getter, URL string, byte array or base64 is read. Denied parts
  are never read.
- After admission the source is sized from its shape, decoded, and attached;
  the reservation shrinks to the exact length. A gate denied only for size
  (too_large/queue_full come after every privacy, identity and sampling
  check), a part larger than the gate, or a data URI of another image type is
  preflighted once more at its exact type and length.
- Unused admitted tickets are released at once. A manager without preflight
  captures nothing. Bedrock bytes keep no adapter cap beyond the manager's.
- Consumed speech bodies are preflighted at read time with the exact size and
  source kind (Bytes or Blob).

Tests: new test/preflight.test.cjs (P-01..P-09), and a sessionLimits option
on the loopback fixture.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ts own reader

SDK design 6.4 lets provider adapters observe a reader the application
obtained. speechResponse now also wraps response.body.getReader(): each chunk
the caller's read() resolves is forwarded to a consumed attachment stream
(startAttachmentStream, 4 MiB reserved), finished when the caller reads to
the end, and aborted on releaseLock, cancel, a read error, a tightened policy
or the cap. The adapter never calls read, clone, tee or getReader itself.
Consumers that never call the public getReader() and Node-stream bodies
still upload nothing. One span is captured once whichever path reads it.

Descriptor: audio.speech.create (openai, groq) adds source consumed_reader.
Tests: new test/speech-stream.test.cjs (S-01..S-07); the media row for a
reader read to its end now expects one upload. The workflow runs both new
test files.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
releaseLock() does not end a ReadableStream: the application can call
getReader() again and read the remaining chunks. The second reader was also
observed, so a read-release-reread sequence uploaded only the tail of the
body as if it were complete (SDK design 6.3 forbids a truncated object).
Only the first reader obtained on the body is observed now; later readers
upload nothing.

Test: S-08 (read two chunks, releaseLock, drain a second reader) expects no
upload and an idle manager.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
nik13 and others added 5 commits November 11, 2026 12:31
…viders

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… adopt the clamped stream reservation

- @traceai/provider-media exports ATTACHMENT_OWNER_KEY (Symbol.for
  "traceai.auto.attachment_owner.v1", the key the auto wire layer and the
  ElevenLabs instrumentation use), attachmentOwnedElsewhere(ctx) and
  ProviderMediaCapture#claim(ctx). A capture built with a parentContext that
  already carries the marker is inert: it never reaches the manager.
- openai audio.* and groq audio.* run the SDK call in a claimed context, so
  wire-level capture skips the call and only the provider adapter uploads it.
  A locally hidden call is still claimed, so the hide holds for every layer.
  Groq chat is not claimed (chat media ownership is unchanged).
- Consumed speech readers rely on the core's min(4 MiB, max_upload_bytes)
  reservation, so servers below 4 MiB now admit them; over-cap bodies are
  discarded whole.
- Tests: owner.test.cjs (O-01..O-05), speech-stream S-09/S-10, and real
  OpenAI/Groq SDK rows that read the marker from the SDK's own fetch context.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…viders

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…lls only

The latest OpenAI SDK makes one extra non-provider request through the custom
fetch on a non-default base URL. The row now checks the audio call by route and
that no request on that host carries the owner marker.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… when it is built alongside

With the openai/anthropic adapter in the same build (stacked lanes), a typed
chat image is captured once by that adapter. The speech and owner rows now
expect exactly that one upload, and still fail on a second copy.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
nik13 and others added 3 commits November 12, 2026 09:02
With provider-media active, an OpenAI speech call is claimed for this layer,
so the auto wire layer skips it. The speech body was observed only through
arrayBuffer()/blob() and the public getReader(). `for await` and values()
lock the stream with an internal reader, so an iterated body uploaded
nothing, while auto alone uploads it once.

Observe values() and [Symbol.asyncIterator] on the body instance through the
same consumed-stream path as a reader: next() chunks are forwarded after the
application's own promise fulfils, done finishes the stream, and return()
(break), a rejected next(), a non-byte chunk or an over-cap body abort it so
nothing partial uploads. Only the first reader or iterator on a body is
observed. A locked body still throws the host error unchanged. Nothing is
read eagerly.

Tests S-11..S-17 cover for-await and values() to the end, break, preventCancel
followed by a reader, the cap, a mid-iteration error, privacy modes and a
locked body.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…s only

The async-iteration hooks also matched node-fetch 2 bodies (a Node
PassThrough has [Symbol.asyncIterator]), so `for await` over an openai 4.x
speech body on the default Node transport uploaded a copy, contrary to the
documented contract and unlike the auto wire layer, which captures through
fetch only. Chunk-wise observation is now gated on a WHATWG body
(getReader present); whole-body arrayBuffer()/blob() reads are unchanged.

Tests: S-18 (Node PassThrough body: no hook, end and break upload nothing)
and an openai 4.x node-fetch shim row in the real-SDK suite (4.52.7 /
4.104.0; for await to the end and break upload 0; arrayBuffer() uploads 1
when enabled or hideInputs, 0 when disabled or piiRedaction).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…uploads

node-fetch 2 resolves blob() to its own Blob class rather than the global
Blob, so provider-media does not observe that read. The openai 4.x
node-fetch row now asserts it (0 uploads, no open stream), and the
speechResponse docstring says so.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant