Skip to content

feat(core): TypeScript attachment streaming, recordings, imports and lifecycle - #677

Draft
nik13 wants to merge 9 commits into
devfrom
feat/attachments-ts-core
Draft

nik13 wants to merge 9 commits into
devfrom
feat/attachments-ts-core

Conversation

@nik13

@nik13 nik13 commented Nov 11, 2026 •

Copy link
Copy Markdown
Contributor

Part of TH-2379 (Linear).

What does this PR do?

Implements the TypeScript half of the shared SDK attachments design (streaming, recordings, imports and lifecycle) in @traceai/fi-core, on top of the opt-in attachment foundation already in this branch. Python (#676) implements the same semantics. Draft component; integrated backend/product qualification has not run.

Defaults are unchanged: attachments are OFF unless explicitly enabled, creating new recordings is separately OFF, and every deny-wins privacy rule (hide flags, PII redaction without a qualified sanitizer, server policy) is evaluated before any source is read.

  • Preflight (manager.preflight(target, {mimeType, direction, maxBytes, ...})): the full deny-wins check without touching the source. It returns a single-use ticket that reserves one descriptor and a byte budget, expires lazily (no timers), is re-checked against the current policy at commit, and can only narrow. Adapters decode or read only after ticket.admitted.
  • Live PCM recordings (startRecording): s16le PCM is cut into independently playable WAV segments on a sample-count clock (at most 30 s and 8 MiB including the 44-byte header, or the negotiated lower limits). All tracks share cut points; track_id appears only for separate_tracks. Segments upload strictly in order with a window of 2; gaps (queue_full, bootstrap_pending, policy_denied, truncated, incomplete, source_unavailable) are recorded with exact durations. Manifests are cumulative and prefix-monotonic, validated locally against the F01 cross-field rules, and authorized only by the latest completed segment's write capability. If the server refuses that after complete (BR-7), remaining segments switch to init → content → manifest → complete.
  • Consumed streams (startAttachmentStream): one bounded upload of chunks the application already read. The core never pulls, clones or tees a stream. Over-cap streams discard everything (too_large); abandoned streams are swept lazily or released by a finalizer; the reference is projected only at finish().
  • Owned recording files (createRecordingFileScope from ./attachments/node, then attachOwnedRecording): positional reads from our own handle, identity bind on (dev, ino, size), change detection before every slice. Tiers: one upload up to 50 MiB; PCM WAV above that (or with known channel roles) becomes a segmented recording built from lease ranges without materializing the file; larger encoded files are too_large. The residual TS path race (an intermediate directory swapped between lstat and open) is documented in the source and covered by a negative test. Windows is unsupported.
  • Completed-recording import (importCompletedRecording): resolver seam (default unavailable), submitted with the original credential and never a write capability, im-{recording} idempotency key, no URL, path or credential in the body, one status GET per call.
  • Renewal: sessions renew near expiry (expires_at - now <= max(60 s, 20% lifetime)) by replaying bootstrap with the same session id and nonce (bs-{sid}-{n}). The check runs from every entry point (preflight, attach, startRecording, attachOwnedRecording, importCompletedRecording), from flush(), status() and worker wake, and from recording writes: tryWrite schedules a deduped check on a microtask, so it does no I/O on the caller's stack. A recording-only workload (tryWrite + flush, as Pipecat-style adapters drive it) therefore renews on its own; upload capabilities renew by replaying the byte-identical initialize (up-{occurrence}). A server that does not mint a fresh capability yields expired and an abort, never a new upload id.
  • Wire adapter (createAttachmentHttpTransportV1): one module for every route, key and status mapping. 404 → not_found, 409 → conflict, Error.code refines only within its HTTP class, Retry-After capped at 5 s, separate control and content deadlines, a best-effort abort after any post-initialize failure, strict JSON (duplicate members rejected), 64 KiB success and 4 KiB error bodies. validAttachmentWire hand-validates every F01 shape the SDK sends or parses.
  • Lifecycle: createNodeAttachmentManager returns DisabledAttachmentManager when OFF. It has the full public surface and starts no timer, socket, process listener, finalizer or bootstrap, and writes no span attribute. Enabled background timers and sockets are unref'd; shutdown finishes open recordings within the budget and returns within the timeout even when a transport ignores its abort signal.
  • Carrier stamp: all four F01 span attributes are written only on the target span of an admitted capture while it is still recording, never on bystander spans.

The legacy four-method AttachmentTransport keeps working for single uploads through an adapter. Since f0a56b37 the adapter is loud: one DeprecationWarning (FI_ATTACHMENTS_LEGACY_TRANSPORT) per process, and startRecording, WAV-tier attachOwnedRecording and importCompletedRecording return unsupported_version at the call instead of uploading segments that can never be bound.

Behaviour changes for stacked adapter lanes (required by the design)

All of these are listed, with how each one now fails and the fix, in typescript/packages/fi-core/ATTACHMENTS-MIGRATION.md, which ships in the package.

  • status().state is "ready" after bootstrap, never "queued". Checks for !== "queued" must change: traceai_auto/src/wire/attachments.ts:60, traceai_agent_core/src/attachments.ts:63, traceai_cursor_sdk/src/attachments.ts:32 (and traceai_auto/test/attachments.test.ts:40).
  • HTTP-level test doubles that mint 60 s sessions see one renewal bootstrap; mint at least 1 h sessions. Since f054a68e, status(), flush(), worker wake and recording writes can also trigger that renewal, not only admission calls.
  • failed is retired (incomplete / unsupported_version) and removed from AttachmentCompletion, so a leftover === "failed" is TS2367 like "queued". manager.correlation(span) no longer stamps attributes.
  • A response that fails its F01 shape check (for example a status body without reason) is unsupported_version and warns once (FI_ATTACHMENTS_WIRE_REJECTED) with the shape name only.
  • A wrapper around createAttachmentHttpTransport(...) that does not forward [ATTACHMENT_TRANSPORT_V1] becomes a legacy transport and triggers the deprecation warning. The feat(coding-agents): opt-in typed image attachments for coding-agent integrations #680 agentAttachmentRuntime guard is that shape today.

Additions in 01b3327a (cross-lane asks; additive, no breaking API)

  • Deferred targets for late-bound spans (manager.deferredCorrelation({ bindTimeoutMs }) returns an AttachmentDeferredCorrelation, which is a subclass of AttachmentCorrelation). Some wire spans are emitted only after the body completes, such as wire-only FormData (OpenAI transcriptions) and wire-only incremental speech (Deepgram speak). preflight, attach and startAttachmentStream now admit Bytes, Blob and consumed-stream captures against a target that is not bound yet, using the full deny-wins check.
    • Before bind, committed bytes stay in the owned budget, capped at half of it. Nothing is read beyond what the caller already handed over: a Blob is retained, not read. Nothing is sent.
    • target.bind(span) checks policy, the environment kill switch and sampling again. Only then does it build the request (with the bound parent and the current policy version), queue the upload and project the reference onto that span.
    • If the bind timeout passes (default 60 s, max 300 s, checked lazily with no timer), target.cancel() is called or the manager shuts down, pending captures end correlation_unavailable and upload nothing.
    • Owned files, recordings and imports never defer.
    • If the span already has a correlation, the bound target forwards to it, so the span keeps one reference list.
    • Disabled managers return an inert target.
  • Consumed-stream reservation is now min(maxBytes, negotiated single-upload cap). On a server with max_upload_bytes below 4 MiB, the 4 MiB stream reservations in ElevenLabs (feat(voice): TS LiveKit Agents recording, Pipecat recording link, ElevenLabs speech attachments #685), provider-media (feat(providers): typed provider media capture for TypeScript Gemini, Bedrock, compatible chat and speech #683) and auto (feat(auto): attachments in @traceai/auto (A02 ownership, A03 wire JSON/FormData, A04 consumed audio) #687) are admitted again, and a body over the effective cap is discarded whole (too_large). preflight keeps exact semantics.
  • Fix: shutdown() no longer overwrites a prior invalidate(reason) with incomplete.
  • Documented in ATTACHMENTS-MIGRATION.md under "Additions (non-breaking)".

Qualified on a temporary private GCP VM (deleted afterwards), loopback only, synthetic data:

Additions in 930876ea (review fixes for the deferred target)

  • Sampling at bind covers every capture on the target. Before, only captures already parked at bind were checked. Now streams still open at bind and tickets preflighted before bind but committed after it also end unsampled when the bound span is unsampled or not recording. Their reserved bytes go back to the owned budget at bind, and the ticket's source is never read.
  • A late bind() fails closed. AttachmentDeferredCorrelation.bind asks the manager whether the bind deadline has passed before it binds the identity. If it has, bind returns false, binds nothing and cancels the target, so open streams and outstanding tickets end correlation_unavailable. This holds even when no other manager entry point ran between the deadline and the bind.
  • Backstops: ticket commit and stream finish re-check the bound span's sampled flag before anything is enqueued. deferredBound also refuses a late target's open streams and tickets if the pre-bind check is bypassed.
  • New @internal host hook deferredExpired(target) (optional on AttachmentDeferredHost). No public API change.
  • Adopter note (feat(openai,anthropic): capture typed images, documents and computer screenshots as attachments #682, feat(auto): attachments in @traceai/auto (A02 ownership, A03 wire JSON/FormData, A04 consumed audio) #687 and other deferred-target callers): target.bind(span) returning false now also means "too late, nothing was bound". Callers must not assume true; nothing else is needed, because the target is already cancelled.
  • Rows T-DT-15..18 in attachments-deferred.test.cjs. ATTACHMENTS-MIGRATION.md is updated.

Tested on GitHub Actions at head 930876ea (the private test VMs were unavailable for this round):

Why?

The user asked for first-class trace attachments, including Pipecat and LiveKit audio that plays in the UI, plus images, files and computer-use screenshots, in both auto modules. The voice and auto lanes need streaming recordings, preflight before decode, owned-file handoff and completed imports from the core rather than per-adapter uploaders.

How was it tested?

All runs used a temporary private GCP VM with synthetic data and in-memory or loopback doubles; no local tests, provider calls or real credentials.

Head f0a56b37 (loud stacked-lane breakage), on a new VM (deleted afterwards):

  • Build passed. pnpm run test:attachments (7 files, now including attachments-compat.test.cjs): 117/117 on Node 24.11.1 and 117/117 on Node 18.20.8. Packed installs on OTel 2.0.1 and 2.12.1: 136/136 each (the CI file list plus the compat file). The compat file passed 8/8 in 5 separate runs. Provider lifecycle 19/19; F01 privacy/identity replay exit 0.
  • New rows C-01..C-08: the legacy transport warns once per process with the fix named; there is no warning for V1, for createAttachmentHttpTransport (which carries V1) or when attachments are OFF; a wrapper that drops V1 is detected; legacy recording, WAV-tier owned recordings and imports are denied up front with zero extra transport calls and no resolver call; deny-wins still holds (policy_denied / disabled beat unsupported_version); the legacy session runs to expired; a status body without reason gives unsupported_version plus one FI_ATTACHMENTS_WIRE_REJECTED warning that contains no capability, credential, URL or id; and tsc --noEmit on test/types/attachments-compat.ts against the built declarations consumes every @ts-expect-error (state !== "queued", status === "failed").
  • Attribution: the same compat file on f054a68e sources fails 5/8 (C-01, C-03, C-04, C-07, C-08), and the updated legacy design row fails there (unpublished). C-02, C-05 and C-06 pass on both: they pin behaviour that must not change.
  • Renewal re-proof: the T-RN rows of the design file run against 031178f6 sources fail T-RN-09 and T-RN-10 (7/9). f054a68e and this head pass 9/9 (Node 24 and 18).
  • Browser leaf: the esbuild bundle has 0 node: imports. In a VM context with no process, the legacy transport hits console.warn exactly once, with the code.
  • Stacked lanes on the PR merge shape (lane + core), this head against f054a68e: frameworks feat(frameworks): typed framework media attachments (LangChain, LlamaIndex, Vercel AI SDK, Genkit, Mastra) #679 helper 43/43 on both; provider-media feat(providers): typed provider media capture for TypeScript Gemini, Bedrock, compatible chat and speech #683 30/30 on both; openai/anthropic feat(openai,anthropic): capture typed images, documents and computer screenshots as attachments #682 attachments-providers 44/44 on both. Coding feat(coding-agents): opt-in typed image attachments for coding-agent integrations #680 + this head: tsc reports TS2367 on "queued" (agent_core/src/attachments.ts:63, cursor_sdk/src/attachments.ts:45) and TS2322 on the transport type. These are the loud, pre-existing compile errors that lane must fix; they are not new.

Head f054a68e (renewal fix), on an earlier VM:

  • Build passed. Attachment suites (6 files): 109/109 on Node 24.11.1 and 109/109 on Node 18.20.8, including new rows T-RN-09 (120 s session, 150 s mono recording, 20 ms tryWrite + 1 s flush, no other entry point: renewal happens, 0 writes rejected after 120 s, 15 segments, published, captureComplete true, no gaps) and T-RN-10 (each trigger alone renews: recording write with no segment sealed and no transport call on the caller's stack, flush, status, worker wake).
  • Regression proof: with the base 031178f6 sources, T-RN-09 and T-RN-10 fail (56/58 in the design file), and the review probe A reproduces the bug (renewSession 0, expired at 116 s, 1500/1500 late writes rejected, gaps [110000,5000,incomplete] and [115000,35000,incomplete], partial). On the head, the same probe gives renewSession 2, ready throughout, 1500/1500 late writes accepted, 15 segments, published. Removing any one trigger (touch, pump, flush, status) turns T-RN-10 red on that trigger.
  • Provider lifecycle and F01 contract replay passed. Full core Jest: 2,253 passed, 13 skipped, 1 failed (escaped decode allocation performance, the 200 ms wall-clock test); the base sources fail the same test on the same VM, so it is VM speed.

Earlier qualification at head 031178f6:

  • Build (CJS, ESM, ESNext) passed.
  • Attachment suites (6 files): 107/107 on Node 22.18.0 and 107/107 on Node 18.20.8. Rows covered: T-PF-01/02/04..08/11..15, T-ST-01..20, T-RN-01..08, T-LS-01/02/03/04/05/06/07/08/09/10, T-IM-01..10, T-CS-01..06, T-CH-01..06, T-LC-02..05/12/15/16, T-TR-01..08, and exit probes disabled, enabled-idle, enabled-blocked, enabled-no-shutdown, mid-recording, import-pending (all exit 0 in under 2 s).
  • T-TR-01: every F01 wire fixture at f3b0923c62 for the shapes the SDK sends or parses agrees with the hand validators.
  • F01 privacy replay (attachments-contract.cjs): all 82 privacy and 11 identity rows through attach AND preflight (T-PF-03); 63 exact reasons on both paths, 10 unsupported positive paths and 9 conservative reason differences fail closed, as before.
  • Provider lifecycle: 19/19.
  • Full core Jest: 2,253 passed, 13 skipped, 1 failed: imageCapEscapedDecodePerformance.test.ts (a 200 ms wall-clock threshold, 249-268 ms on the VM). On the same VM the untouched base b5831771 fails the same test 3/3 (249, 258, 253 ms) as the head does (258, 258, 250 ms), so the failure is VM speed, not this change.
  • GitHub CI on the pushed head is the authority for timing suites and runs the packed OTel floor/latest, contract and browser-leaf gates.

E2E coverage

not applicable (no e2e/ harness in this repository) · flows hit: n/a · marker: n/a · verdict: not applicable. Product playback (OBS-E2E-061) needs the backend and ingest stages.

Checklist

  • Base is dev.
  • Conventional commits; synthetic values only.
  • Public TypeScript signatures plus failure-path tests.
  • Capture and new recordings stay opt-in.
  • Integrated backend/ingest/UI qualification (Q01/Q03).
  • Customer-facing documentation.

Notes for reviewers

  • Not covered here: T-PF-09/10 and T-CS-07 belong to the auto, voice and Python lanes; T-LC-01/13/14 and the fork probe are Python-only; T-LS-11 needs a Windows runner.
  • Beyond the design text: in the BR-7 fallback order, a segment holds its complete (and one of the two worker slots) until the next segment is sealed or the recording finishes, bounded at 35 s, so the last capability can still PUT the final manifest. Reconciliation should confirm this.
  • GitHub CI on head f054a68e: 129 checks pass, 0 fail (13 skipped by path filters), including TypeScript attachment core and TypeScript core provider tests.
  • GitHub CI on head f0a56b37: see the checks tab; the TypeScript attachment core workflow runs the new compat file through test:attachments.
  • Backend requirements BR-1..BR-7 and contract questions CQ-1..CQ-5 are implemented with the design's fail-closed fallbacks; none is assumed confirmed.
  • No merge, release or playback claim. Rollback is leaving attachments disabled.

🤖 Generated with Claude Code

nik13 and others added 3 commits November 10, 2026 17:15
Provide deny-wins admission, bounded Bytes/Blob transfers, explicit correlation and provider lifecycle integration. Keep unqualified retrieval and streaming paths closed.

Co-Authored-By: Claude Code <noreply@anthropic.com>
Playwright 1.56.1 does not support the Ubuntu 26.04 image now selected by ubuntu-latest; retain the qualified browser on Ubuntu 24.04.

Co-Authored-By: Claude Code <noreply@anthropic.com>
Keep ready and flush deadlines referenced only while explicitly awaited. Background requests and retries remain unreferenced. Reproduce prior early CLI exit with a real TLS child process that has no artificial keepalive.

Co-Authored-By: Claude Code <noreply@anthropic.com>
nik13 and others added 2 commits November 10, 2026 21:53
…lifecycle per the SDK design (WIP)

Implements sdk-streaming-design sections 0-9 in fi-core attachments: preflight tickets,
PCM recording writer with WAV segmentation and prefix-monotonic manifests, consumed-stream
uploads, completed-recording import seam, owned recording file scope and tiers, capability
and session renewal by replay, the disabled null object, the one versioned wire adapter
(idempotency keys, status table, abort, manifest, import routes) and hand validators for
every F01 shape. The legacy four-method transport keeps working through an adapter.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Type FinalizationRegistry locally so consumers compiling fi-core source with an ES2020
lib still build. Hold a fallback-order segment's complete until a later segment is sealed
or the recording finishes, so the last capability can PUT the final manifest (BR-7).
Add T-ST-16 fallback, T-CS-05 forced-GC, T-TR-04 proxy-canary and T-PF-03 preflight
replay coverage.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@nik13 nik13 changed the title feat(core): add opt-in TypeScript attachment foundation feat(core): TypeScript attachment streaming, recordings, imports and lifecycle Nov 11, 2026
nik13 and others added 2 commits November 10, 2026 23:04
…atus and worker wake

A recording-only workload (tryWrite + flush, as Pipecat-style adapters drive it)
never reached maybeRenew(), which ran only from the five admission entry points.
A recording that outlived the server session lost its tail as `incomplete` gaps.

Per design 3.1, renewal is now also checked from:
- pump() (worker wake),
- flush() and status() (enter() instead of sweep()),
- AttachmentRecording.writeFrames, through a deduped host touch() that defers
  maybeRenew to a microtask so tryWrite does no I/O on the caller's stack.

Regression tests: T-RN-09 (120 s session, 150 s recording, tryWrite + flush only:
renewal, no rejected writes, 15 segments, published, no gaps) and T-RN-10 (each
trigger alone renews: recording write, flush, status, worker wake).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Changes that used to stop capture silently now fail loudly:
- A legacy four-method transport (including a wrapper that drops
  ATTACHMENT_TRANSPORT_V1) emits one DeprecationWarning
  FI_ATTACHMENTS_LEGACY_TRANSPORT naming what it cannot do and the fix.
  startRecording, WAV-tier attachOwnedRecording and
  importCompletedRecording are denied at the call with
  unsupported_version instead of buffering and uploading segments that
  can never be bound. Renewal is no longer retried on a wire that has no
  renewal route.
- A response failing its F01 shape check still fails closed as
  unsupported_version, and now warns once (FI_ATTACHMENTS_WIRE_REJECTED)
  with the shape name only.
- `failed` leaves AttachmentCompletion, so a leftover check is TS2367
  like the retired `queued` ready state. A tsc test pins both.
- ATTACHMENTS-MIGRATION.md lists every change, how it fails, and the fix.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
nik13 and others added 2 commits November 11, 2026 07:33
Adds manager.deferredCorrelation(): Bytes, Blob and consumed-stream
captures can be admitted before their span exists (wire spans that are
emitted only after the body completes). Committed bytes stay in the
owned budget, capped at half of it, and nothing is sent until
target.bind(span), which re-evaluates policy and sampling. Timeout,
cancel() and shutdown fail pending captures with
correlation_unavailable. Owned files, recordings and imports never
defer, and a span that already has a correlation keeps one reference
list.

startAttachmentStream now reserves min(maxBytes, negotiated
single-upload cap), so servers below 4 MiB still admit consumed
streams. shutdown() keeps an earlier invalidate() reason instead of
overwriting it with incomplete.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Bind-time sampling and the bind deadline now cover all captures on a
deferred target, not only those already parked at bind:

- deferredBound aborts streams still open on the target and releases its
  outstanding ticket reservations when the bound span is unsampled or not
  recording (`unsampled`), or when the bind is late (`correlation_unavailable`).
- AttachmentDeferredCorrelation.bind asks the host whether the deadline has
  passed BEFORE binding the identity. A late bind cancels the target and
  returns false, so tickets and open streams can no longer upload after the
  deadline when no other manager entry point ran in between.
- Backstops: ticket commit and stream finish re-check the bound span's
  sampled flag before anything is enqueued.

Rows T-DT-15..18 cover a stream open at bind and a ticket committed after
bind on an unsampled span, a late bind with no intervening entry point, and
each backstop with the primary check bypassed.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant