Skip to content

feat(voice): TS LiveKit Agents recording, Pipecat recording link, ElevenLabs speech attachments - #685

Draft
nik13 wants to merge 10 commits into
feat/attachments-ts-corefrom
feat/attachments-ts-voice
Draft

nik13 wants to merge 10 commits into
feat/attachments-ts-corefrom
feat/attachments-ts-voice

Conversation

@nik13

@nik13 nik13 commented Nov 11, 2026 •

Copy link
Copy Markdown
Contributor

Part of TH-2379 (Linear).

Summary

TypeScript voice attachments for LiveKit Agents, Pipecat and ElevenLabs, built on the shared attachment core in #677 (feat/attachments-ts-core). Attachments stay off unless the provider's manager is enabled. A new recording is never started; these adapters only upload audio that already exists or that the application already read.

Item Package What ships
V04 @traceai/fi-instrumentation-livekit New Node-only subpath /agents with bindLiveKitSessionRecording(session, ctx, { manager, agents, ownedRecordingFiles, finalizationTimeoutMs, capture }). At session end it uploads the audio.ogg that LiveKit Agents' own RecorderIO wrote.
V05 @traceai/fi-instrumentation-pipecat observePipecatClient(client, { serverRecording }) links client event spans to the server recording's trace and occurrence. It records nothing in the browser.
V09 @traceai/elevenlabs Uploads the speech-to-text input Blob, the complete convertWithTimestamps MP3, and the MP3 from convert, stream and streamWithTimestamps that the application reads to the end (consumed-stream writer).

V04: LiveKit Agents (Node) recording

  • No recording control. The binding never sets record, never starts or stops the recorder, and never deletes the file.
  • Chained hook. It wraps the SDK's own JobContext._onSessionEnd and always runs the original afterwards. The order was confirmed by reading the source of 1.8.0 and 1.9.5:
    1. AgentSession.close() closes the RecorderIO;
    2. the application's onSessionEnd runs;
    3. ctx._onSessionEnd() runs, which is where the binding runs, before LiveKit's own report upload.
  • What qualifies. Only the job's primary session, its own closed recorder, and the fixed audio.ogg in ctx.sessionDirectory. The report path is checked, but it does not grant permission on its own.
    • Before handing the file to the core, the binding rejects a symlink, a hard link or a file owned by another user.
    • The core's owned-recording lease then does the upload: it identifies the file, opens it without following symlinks, binds to its identity and checks it on every slice. Tier single, direction mixed, purpose conversation_recording.
  • Deny wins. It uploads nothing in any of these cases:
    • the core's public preflight denies, checked at bind and again at session end;
    • LiveKit project or session redaction is on;
    • LIVEKIT_TELEMETRY_ALLOW_PII is false;
    • session audio recording is off.
  • Supported versions. Exactly 1.8.0, 1.8.1 and 1.9.0 to 1.9.5. Any other version reports unsupported_version.
    • The application passes in the @livekit/agents module, so this package never imports it.
    • There is no peer dependency, so pnpm-lock.yaml is unchanged. Declaring an optional peer made a lockfile refresh add the whole Agents tree.
  • Deadline. The upload is bounded by finalizationTimeoutMs (at most 30 s). A late step after the deadline opens nothing and uploads nothing.

V09: ElevenLabs consumed streams

  • Only chunks the application already pulled through the instrumentation's own zero-prefetch bridge are copied. Nothing is read ahead, teed or drained.
  • The stream finishes before the span ends, so its reference is projected onto the span.
  • Cancel, error, timeout, breaking out early, or going over the cap uploads nothing. The cap is 4 MiB, or the server's max_upload_bytes when that is lower. A partial file is never stored as if it were complete.
  • PCM and µ-law output formats fail closed with source_unavailable.
  • A bound call sets the shared context marker traceai.auto.attachment_owner.v1. Wire-level auto capture that honours the marker will not upload the same audio again.
  • The audio size and latency metadata keep their previous behaviour. The restored work had gated them; I reverted that because it is outside this opt-in.

Additions in fd7ffc41 (merges feat/attachments-ts-core at 930876ea)

  • Merged the updated core (930876ea: deferred targets, clamped stream reservation, unsampled and late-bind fixes). No voice adapter uses deferredCorrelation() or target.bind(), so the new "bind() can return false when late" rule doesn't affect this PR. ElevenLabs attaches to its own span, LiveKit uses manager.correlation(span), and Pipecat only adds a link.
  • ElevenLabs adopts the clamped reservation. A consumed stream asks for 4 MiB, and the core admits it at min(4 MiB, server max_upload_bytes). A server with a limit below 4 MiB now captures the stream (before this, it was denied too_large). A body over the effective cap is still dropped whole (too_large).
  • README concurrency number. The manager reserves the whole cap from its owned budget of about 31 MiB (32 MiB minus transfer scratch). That fits 7 concurrent 4 MiB streams, and the 8th reports queue_full. The README used to say "many".
  • LiveKit Agents session end: policy before any file access. At session end the binding re-checks the manager policy right after the native LiveKit checks, before it builds the SDK report and before any stat or open. Previously the check came after the report and before the lstat.
    • This is defense in depth. No supported Agents version touches the file while building the report, which is why mutation M6 (the old position) survives.
    • README: the recording must be owned by the worker's own user, and every check runs again at session end before the file is touched.
  • New tests.
    • ElevenLabs, server limit of 1 MiB: convert and stream upload on the floor SDK and the current SDK. With an exact 1 MiB body, one copy uploads; one byte more is too_large.
    • ElevenLabs, concurrency: 7 streams at 4 MiB are open and the 8th is queue_full. Under a 1 MiB limit, 16 streams are admitted.
    • LiveKit, session end: a watch records every fs/fs/promises call in the session directory while the binding finalizes. A control row proves the watch sees the lstat and open. Five ways of tightening policy after bind each deny with zero file calls, both with the file present and with it removed:
      • PII redaction;
      • outputs hidden;
      • manager disabled;
      • the env kill switch;
      • invalidate.
    • LiveKit, file ownership: a recording chowned to another uid (needs root; CI runs the suite as root) is rejected source_unavailable. So is a recording whose owner differs from the worker's uid (runs without root). Nothing is uploaded and the file is kept.

Testing for these additions (private GCP VM, loopback-only networking, VM deleted)

The VM ran the fd7ffc41 tree: 31409b33 with the fd7ffc41 test fix applied in place. The two trees are otherwise byte-identical.

  • ElevenLabs: 155/155 on Node 22 and on Node 20.19. Lint and the type contract pass.
  • LiveKit Agents, real SDK: 40/40 on each of the 8 supported versions (1.8.0, 1.8.1, 1.9.0 to 1.9.5), running as root. As a non-root user, 39 pass and 1 is skipped: the chown row skips and the uid-mismatch row passes.
  • Pipecat and LiveKit jest: Pipecat 262/262 on Node 22 and on Node 18; LiveKit jest 26/26.
  • Attribution against 457a1553 (old core f0a56b37):
    • all 6 new ElevenLabs tests fail there, and they pass on this head;
    • the new LiveKit tests pass there too. They are regression guards for behaviour 457a1553 already had (deny before stat, uid check), and the mutations below prove they guard it.
  • Mutations:
    • 5 of 6 detected:
      • M1, the policy check moved after the lstat: 10 rows fail;
      • M2, no policy check at session end: 10 rows fail;
      • M3, uid check removed: 2 rows fail;
      • M4, core clamp removed: 6 rows fail;
      • M5, adapter cap doubled: the concurrency row fails.
    • M6, the check at its old position after the SDK report, survives as explained above.
  • Flake check: the new rows passed 3 times in a row.
  • CI on fd7ffc41: ElevenLabs, Pipecat, LiveKit (including Agents recording on 1.8.0 and 1.9.5, 40/40 each) and the TS attachment core all pass (dispatched; the workflows filter on PRs into dev/main).
    • The first LiveKit run, on 31409b33, failed 5 rows: the test's own rmSync (which calls lstat) ran inside the file watch. fd7ffc41 starts the watch after the removal.

Testing at 457a1553 (private GCP VM, loopback-only networking; no network access beyond the loopback interface)

  • LiveKit Agents, real SDK, at all 8 supported versions (1.8.0, 1.8.1, 1.9.0 to 1.9.5): 27/27 each.
    • The SDK's own RecorderIO writes a real Ogg/Opus audio.ogg and the real JobContext._onSessionEnd runs.
    • The uploaded bytes are identical to the file, and ffprobe decodes them as opus.
    • Negative controls:
      • attachments off;
      • env kill switch;
      • core PII, hide input and hide output;
      • server redaction and server-denied owned files;
      • LiveKit redaction and LIVEKIT_TELEMETRY_ALLOW_PII=0;
      • policy tightened after bind;
      • recorder still recording;
      • recorder absent;
      • symlinked, hard-linked or deleted file;
      • recorder path outside the session directory;
      • another primary session;
      • T-LS-05 file swap after identify;
      • stalled transfer hits the deadline;
      • idempotent bind and conflict;
      • unsupported version.
  • ElevenLabs: 149/149 on Node 22 and on Node 20.19 (the engines floor), with both the floor and the current SDK. Lint and the type contract pass.
  • Pipecat: 262/262 on Node 18.20.8 and Node 22, with the floor (0.3.0) and current (1.16.0) client. The customer example typechecks.
  • LiveKit jest unit suite: 26/26.
  • Mutation negative controls: 12 of 13 detected (each mutation must make its targeted suite fail).
    • Covered: stream finished after the span ends; cancel finished instead of aborted; hide flags ignored; PCM labelled MP3; owner marker dropped; LiveKit redaction ignored; hard links accepted; SDK session-end step skipped; no re-check at finalization; PII kill switch ignored; any version accepted; Pipecat conversation id unchecked.
    • The one survivor removes the binding's own isFile check. The core's no-follow open still rejects the symlink, so this is a layered check and the test still passes.
  • CI (dispatched on this branch at head 50cb9956): ElevenLabs, Pipecat, LiveKit (including Agents recording on 1.8.0 and 1.9.5, 27/27 each) and the TS attachment core are all green.

CI: the workflows filter on PRs into dev/main, so I dispatched them on this branch. livekit-typescript-tests.yml gains workflow_dispatch and a new agents-recording job (floor 1.8.0, latest 1.9.5).

Not done (fails closed)

  • LiveKit Egress completed-recording import from TS. The core seam (importCompletedRecording) exists, but no qualified Egress output reader or registry exists (backend B08 READERS is empty; CQ-4). It stays source_unavailable.
  • WAV-tier LiveKit recordings. The SDK always writes Ogg/Opus, so only tier single applies. Files above the single-upload cap are too_large.
  • Raw PCM and µ-law ElevenLabs output. These need startRecording with a known sample format. Not wired; source_unavailable.
  • Product playback (Q03/Q04). No real recorded call has been played back through Future AGI yet. That needs backend B06/B08/B11, ingest I04 and UI U02 integrated.
  • Auto wire dedup. It relies on the auto lane honouring the shared owner marker. Auto attachment capture is not on this base.
  • Pipecat browser recording. Out of scope by design (reference only).

🤖 Generated with Claude Code

nik13 and others added 6 commits November 11, 2026 01:35
…attachments

Restores the ts-voice lane's saved work (refs/recovery/att-b-voice-1111)
onto the current TS attachment core:
- Pipecat client events link to an authenticated server recording
  occurrence (reference only, no browser capture).
- ElevenLabs STT Blob input and complete timestamped MP3 output attach
  through the shared provider manager.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…am writer

- convert, stream and streamWithTimestamps feed only chunks the
  application already read (instrumentation-owned bridge) into
  startAttachmentStream; nothing is read ahead, teed or drained.
- The stream finishes before the span ends so its reference projects;
  cancel, error, timeout, early break or over 4 MiB uploads nothing.
- Non-MP3 output formats fail closed with source_unavailable.
- Bound calls carry the shared media-owner context marker so wire-level
  capture can skip the same bytes.
- Size and latency metadata keep their previous behaviour; test doubles
  mint 1 h sessions.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
New Node-only subpath @traceai/fi-instrumentation-livekit/agents:
bindLiveKitSessionRecording(session, ctx, { manager, agents,
ownedRecordingFiles, finalizationTimeoutMs, capture }).

- Never enables, starts, stops or deletes a recording. Chains the SDK's
  own JobContext._onSessionEnd, which runs after AgentSession.close()
  closed the RecorderIO and after the application's onSessionEnd, and
  always runs the SDK step afterwards.
- Only the job's primary session, its own closed recorder and the fixed
  audio.ogg in the job's session directory qualify; the file goes
  through the core's owned-recording lease (identify, no-follow open,
  identity bind, per-slice check) as tier single, direction mixed.
- Deny wins: core policy via public preflight at bind and again at
  finalization, LiveKit project/session redaction, the
  LIVEKIT_TELEMETRY_ALLOW_PII kill switch and session audio off.
- Exact supported versions 1.8.0, 1.8.1, 1.9.0-1.9.5; others report
  unsupported_version. No package peer, so the lockfile is unchanged.
- CI job runs the real SDK (floor 1.8.0, latest 1.9.5) with its real
  RecorderIO writing Ogg/Opus; the workflow gains workflow_dispatch.
- Pipecat: a throwing server-identity callback reports
  source_unavailable without interrupting client callbacks.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ed assertion

A failing assertion used to leave the SDK RecorderIO running, which kept
the test process alive until the job timeout.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The owned-file scope requires a canonical root; macOS's default temp
directory sits under the /var symlink, so it fails closed there.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
nik13 and others added 4 commits November 11, 2026 17:06
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…nd deny order

- ElevenLabs: consumed streams now rely on the core's min(4 MiB, server
  single-upload limit) reservation, so servers whose max_upload_bytes is
  below 4 MiB admit them. Tests cover a 1 MiB server (real SDK floor and
  current), the exact-limit / limit+1 boundary, and the owned-budget
  concurrency (7 open 4 MiB streams, the 8th queue_full). README states
  the real concurrency instead of "many".
- LiveKit Agents binding: the session-end policy re-check now runs first,
  before the SDK report is built and before any stat or open. New tests
  watch the session directory's fs calls and assert none happen before a
  deny (five tightening paths, with and without the file), plus a control,
  and reject a recording owned by another user.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…watch

rmSync lstats the file it removes, so watching from before the removal
recorded the test's own call. The watch now starts right before the
session end it observes.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant