Skip to content

feat(frameworks): typed framework media attachments (LangChain, LlamaIndex, Vercel AI SDK, Genkit, Mastra) - #679

Draft
nik13 wants to merge 11 commits into
feat/attachments-ts-corefrom
feat/attachments-ts-frameworks
Draft

nik13 wants to merge 11 commits into
feat/attachments-ts-corefrom
feat/attachments-ts-frameworks

Conversation

@nik13

@nik13 nik13 commented Nov 11, 2026 •

Copy link
Copy Markdown
Contributor

Part of TH-2379 (Linear).

Draft. Stacked on #677 (feat/attachments-ts-core). Design work packages F03 to F06.

Resync pass 2026-11-11 (head 2b38d3f0)

  • Merged the updated feat/attachments-ts-core (f0a56b37), as merge commit 47614991. The merge was clean. Before any adaptation, the merged tree already passed: the helper suite (43/43) and all 8 real-framework floor/latest rows, with unchanged upload counts. Capture never stopped on the new core.
  • The helper now uses the core preflight API (SDK design section 1). captureMediaParts sizes each part from its shape: the exact decoded length, computed without decoding. It then calls manager.preflight(span, {mimeType, direction, purpose, capture, maxBytes}). Only an admitted part is decoded and handed to ticket.attach(...), with its message and part fields. The lazy get data() getter through manager.attach is gone.
    • A part that does not fit the owned-byte budget is now denied as queue_full before any decode. The old path decoded inside the core first, then denied.
    • When an admitted part fails strict decoding, its ticket is released at once. Decoding no longer throws, so the later parts of the message still run.
    • Uint8Array parts are not decoded, so only the manager bounds them at preflight, exactly as before. Base64 strings keep the 1 MiB per-part decode cap. Commit 2b38d3f0 fixes 47d6b724, which had wrongly applied the 1 MiB cap to bytes too, so that a 1.5 MiB typed-bytes image was skipped.
    • The public API (captureMediaMessages, captureMediaParts) and every binding are unchanged.
  • The stream API is not adopted, and that is deliberate. Every framework boundary hands over each media part as one complete inline value; streamed AI SDK 7 files arrive whole in the end event. No consumed byte stream exists here, so startAttachmentStream does not apply.
  • New helper tests:
    • queue_full denial with zero atob calls, for all four dialects;
    • preflight + ticket.attach are used and manager.attach is never called;
    • a ticket is released on a decode failure, and the next part still uploads;
    • the fixture's transport emits no FI_ATTACHMENTS_* warning, which pins it to the v1 wire.
    • a 1.5 MiB Uint8Array image uploads byte-identical, while the same media as base64 in the same array is skipped.
  • GCP VM qualification (two private VMs, both deleted; synthetic data, loopback attachment API, network-isolated test processes). Final round on 2b38d3f0:
Run Result
Helper suite on the head, Node 22.23.4 and Node 24.11.1 48/48 each, 0 FI_ATTACHMENTS_* warnings
Real framework invocations on the head, Node 22 and Node 24 8/8 rows each
Same, merged core without adaptation (control) helper 43/43, framework rows 8/8
The head's tests on the control build 43/45
The head's tests on 47d6b724 44/45
Existing suites vercel 186/186, genkit 141/141, mastra 156/156; llamaindex jest finds no tests (exit 0)
Mutation: decode before preflight 25 rows fail
Mutation: no ticket release 1 row fails
Mutation: back to the manager.attach getter 2 rows fail
  • The real framework invocations cover @langchain/core 0.2.0 and 1.3.7, llamaindex 0.1.0 and 0.12.1, ai 5.0.0 and 7.0.169, and genkit 1.42.0 and 1.44.0. Allowed runs upload 1, 1, 3 (AI SDK 5) or 5 (AI SDK 7), and 5 parts respectively, the same counts as the control. Every denied mode uploads 0.

  • On the control build, the 2 failing rows are exactly the two preflight rows; the large-bytes row passes there, because the old helper captured it. On 47d6b724, the 1 failing row is exactly the large-bytes row.

  • CI at 2b38d3f0: all 21 PR checks pass, including TypeScript framework attachments on Node 22 and 24 (run 43020780762). Branch dispatches, which do not appear in this PR's Checks tab, all pass:

    • Zero-code agent (@traceai/auto) tests: 43020906548
    • LangChain TypeScript contracts: 8/8 jobs, 43020912578
    • Vercel AI SDK native telemetry tests: 6/6, 43020918123
    • Mastra TypeScript tests: 4/4, 43020923351
    • TypeScript framework attachments: 2/2, 43020928934
  • Still open: cross-span occurrence claim. A framework integration and a provider integration given the same manager still upload the same media once per span.

Fix pass 2026-11-11 (head 4e24a9ce)

  • Auto registry (P1). The helper lived at typescript/packages/traceai_framework_attachments. The @traceai/auto registry test (traceai_auto/test/config.test.ts) requires every packages/traceai_* directory to be a registered instrumentation, so it failed: expected 52, received 51. The helper is not an instrumentation, so it now lives at typescript/packages/framework_attachments. The npm name stays @traceai/framework-attachments.
    • The release-please config and manifest keys, repository.directory, the tsconfig references in langchain, llamaindex, vercel and genkit, the lockfile importer, and the workflow path filters and pack loops all moved with it.
    • Regression test: framework_attachments/test/layout.test.cjs. It fails if the helper re-enters the traceai_ prefix, or if any release key, consumer tsconfig reference or lockfile link still names the old directory. Copied into the old tree at 6aa72cb2, it fails as expected (2 of 3 pass); at the new head, 3 of 3 pass.
  • Base drift. feat/attachments-ts-core moved to 031178f6, and its v1 transport now rejects status bodies that have no reason member. The helper's loopback fixture now returns reason, and its sessions and uploads expire an hour out. This is a test-only change, and both the old and the new core accept it.
  • Proof on temporary private GCP VMs (Node 22.18, synthetic data, both VMs deleted and confirmed gone). Two trees were tested: the branch head 2ada9bc1 on the old core, and the PR merge tree (4e24a9ce + 031178f6). Both gave the same results:
    • frozen installs (filtered and full) leave the lockfile unchanged, and pnpm run build:ordered passes;
    • auto config.test.ts passes 12/12, and the full @traceai/auto suite passes 44/44 suites (1504 passed, 12 skipped);
    • all 43/43 helper tests pass;
    • real framework invocations pass at floor and latest for LangChain, LlamaIndex, AI SDK and Genkit (8 of 8);
    • the llamaindex, vercel (186), genkit (141) and mastra (156) suites pass, and the packed Mastra contract passes for CJS and ESM;
    • LangChain plain jest passes 74/74 tests. Only the pre-existing legacyFunctionArguments.test.ts type error fails, and it fails the same way on base 7145f783.
  • CI at 4e24a9ce, all green:
    • pull_request TypeScript framework attachments (merge ref);
    • dispatched on the branch: Zero-code agent (@traceai/auto) tests, TypeScript framework attachments, LangChain TypeScript contracts, Vercel AI SDK native telemetry tests and Mastra TypeScript tests. Run ids: 42973681302, 42973842136, 42973850962, 42973859184, 42973868102, 42973875575.

What this adds

  • @traceai/framework-attachments is a shared typed-part normalizer. It never uploads anything itself; the shared AttachmentManager from feat(core): TypeScript attachment streaming, recordings, imports and lifecycle #677 owns admission, transport, references and lifecycle.
    • Decoding is lazy and runs inside the manager's admission check. Disabled, policy-denied, PII-redacted, hidden and unsampled captures never decode or copy media.
    • Sources are inline base64, data URIs and Uint8Array only, capped at 1 MiB per part and 64 positions per span.
    • No URL fetches, file paths or stream reads.
  • Bindings (all opt-in through attachments; capture restrictions can only add denials):
    • LangChain LangChainInstrumentation({attachments, capture}) covers chat-model input and output messages.
    • LlamaIndex LlamaIndexInstrumentation({attachments, traceConfig, capture}) covers chat input and non-streaming output.
    • Vercel AI SDK VercelAIInstrumentation({attachments, capture}) covers prompt messages on AI SDK 5, 6 and 7. Model response files and typed tool content results are covered on AI SDK 7 only. recordInputs and recordOutputs set to false also deny the matching media.
    • Genkit has three bindings: genkitAttachmentMiddleware(binding) for any model, including plugin models, withGenkitModelAttachments and withGenkitToolAttachments.
    • Mastra captures no raw media (N). The exporter sees spans only after they end, has no manager and never uploads. It also removes fi.attachment.* attributes it did not issue.
  • The per-operation E/R/N table is in typescript/packages/framework_attachments/README.md.

Not supported, fails closed

  • Imports and recordings: not applicable to these framework boundaries. Byte streams: none exist at these boundaries (see the resync pass above).
  • Remote and provider retrieval.
  • LlamaIndex streaming output, AI SDK 5 and 6 outputs, and LangChain chain, tool and retriever objects.
  • Cross-span deduplication. If both a framework integration and a provider integration (for example LangChain plus @traceai/openai) get the same manager, the same media is uploaded once per span. The core has no occurrence-claim API yet. This was raised for reconciliation.

CI and packaging

  • New workflow framework-attachments-tests.yml, triggered on any pull request that touches these paths.
  • The Vercel and Mastra consumer installs, and LangChain's README and strict-pnpm installs, now include the candidate helper tarball.
  • Vercel and Mastra workflows gained workflow_dispatch, because their pull_request triggers only fire for dev and main.
  • The helper is registered with release-please at 0.1.0. Nothing is published.

Evidence (GitHub Actions on this branch; the GCP VM run could not start because gcloud needed interactive reauth)

  • TypeScript framework attachments, Node 22 and 24:
    • 40/40 helper tests: privacy negatives for disabled, hide flags, PII and server redaction policy, with no decode and no upload; lifetime and occurrence checks.
    • Real framework invocations against a loopback attachment API at floor and latest: @langchain/core 0.2.0 and 1.3.7, llamaindex 0.1.0 and 0.12.1, ai 5.0.0 and 7.0.169, genkit 1.42.0 and 1.44.0.
    • Each was run as allowed, disabled, hide, PII and native-content-disabled. Allowed uploads byte-identical media (LangChain 1, LlamaIndex 1, AI SDK 3 on 5.x and 5 on 7.x, Genkit 5); every denied mode uploads 0.
    • AI SDK streaming cancellation passes.
    • Existing llamaindex, vercel, genkit and mastra suites pass.
    • Mastra attachment contract (packed, real Agent over loopback, CJS and ESM) passes.
  • The following were dispatched on this branch at 6aa72cb2. They are branch runs, so they do not show up in this PR's Checks tab:
  • Pull-request workflow TypeScript framework attachments: https://github.com/future-agi/traceAI/actions/runs/42960681184
  • Known and out of scope: running the LangChain package's plain jest hits a TS type error in src/__tests__/legacyFunctionArguments.test.ts, a file this PR does not touch. LangChain's own workflow, which runs its consumer runtime suites, is the gate used here.

🤖 Generated with Claude Code

nik13 and others added 8 commits November 10, 2026 20:03
…tachment manager

Adds @traceai/framework-attachments, a shared typed-part normalizer with
thin bindings for LangChain LLM runs, LlamaIndex chat, Vercel AI SDK
model/tool events and Genkit model/tool implementations. Decode is lazy
behind manager admission, inline base64/data URIs and Uint8Array only,
capped at 1 MiB per part. No URL, path or stream reads.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… media CI

Adds genkitAttachmentMiddleware for models the application does not
implement. The Mastra exporter removes fi.attachment.* keys it did not
issue and documents that it never captures media. Adds a pull-request
workflow for the framework media matrix, packs the shared helper in the
Vercel and Mastra consumers, and registers it with release-please.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…its own consumer suites

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…taller uses candidate transitive tarballs

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…e tarball in the strict pnpm load

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…rted input

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…trumentation prefix

The @traceai/auto registry test requires every packages/traceai_* directory to
be a registered instrumentation. @traceai/framework-attachments is a shared
helper with no instrumentor, so its directory moves to
packages/framework_attachments (npm name unchanged). Release-please keys,
repository.directory, consumer tsconfig references, lockfile importer and
workflow path filters and pack loops follow the move.

Adds test/layout.test.cjs, which fails if the helper re-enters the traceai_
prefix or if any release key, tsconfig reference or lockfile link still points
at the old directory.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The TypeScript core base (feat/attachments-ts-core 031178f) now validates
content and complete status bodies strictly ({id, state, reason}). The
helper's loopback fixture returned {upload_id, state} only, so the PR merge
ref failed five upload tests. Status bodies now include `reason`, and session
and upload expiry sit an hour out so no renewal runs inside a test. Both the
previous and the current core accept this fixture.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
nik13 and others added 3 commits November 11, 2026 03:43
…meworks

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Adopt the ts-core preflight API (SDK design section 1) in the shared
framework media helper used by the LangChain, LlamaIndex, Vercel AI SDK
and Genkit bindings.

- Size each part from its shape only (exact decoded length, no decode),
  then manager.preflight() with the merged capture policy. Only an
  admitted part is decoded and passed to ticket.attach() with its
  message/part fields. The lazy `get data()` getter through
  manager.attach() is gone.
- A part over the owned-byte budget is now denied queue_full before any
  decode (the old path decoded inside the core, then denied).
- An admitted part that fails strict decode releases its ticket at once;
  decode no longer throws, so later parts in the same message still run.
- Tests: queue_full denial with zero atob calls for all four dialects,
  preflight+ticket.attach used and manager.attach never called, ticket
  release on a decode failure, and no FI_ATTACHMENTS_* warning from the
  v1 fixture transport.

No consumed byte stream exists at these boundaries (every part arrives
as one complete inline value), so startAttachmentStream is not used.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…e base64 cap

The preflight rewrite applied the 1 MiB base64 decode cap to Uint8Array
parts too, so a typed-bytes image over 1 MiB (for example an AI SDK
`image` part holding an uploaded photo) was skipped where the previous
helper captured it. Bytes are never decoded here: size them by
byteLength and let preflight apply the manager's upload cap and
owned-byte budget. Base64 strings keep the 1 MiB decode cap.

Test: a 1.5 MiB Uint8Array part uploads byte-identical, and the same
media as base64 in the same array is skipped.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant