Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
38 changes: 38 additions & 0 deletions .github/workflows/livekit-typescript-tests.yml
Original file line number Diff line number Diff line change
Expand Up @@ -9,6 +9,7 @@ on:
- 'typescript/packages/fi-semantic-conventions/**'
- 'typescript/pnpm-lock.yaml'
- '.github/workflows/livekit-typescript-tests.yml'
workflow_dispatch:

permissions:
contents: read
Expand Down Expand Up @@ -85,3 +86,40 @@ jobs:
env "LIVEKIT_COMPAT_DIR=$RUNNER_TEMP/livekit-sdk" \
"LIVEKIT_INSTRUMENTATION_DIR=$RUNNER_TEMP/livekit-sdk/node_modules/@traceai/fi-instrumentation-livekit" \
"$(command -v node)" --test packages/traceai_livekit/test/readme-shutdown.cjs

agents-recording:
name: LiveKit Agents recording binding (${{ matrix.agents }})
runs-on: ubuntu-24.04
timeout-minutes: 15
strategy:
fail-fast: false
matrix:
# Floor and latest of the exact versions the binding supports.
agents: ['1.8.0', '1.9.5']
defaults:
run:
shell: bash
working-directory: typescript
steps:
- uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683 # v4.2.2
with:
persist-credentials: false
- uses: actions/setup-node@49933ea5288caeca8642d1e84afbd3f7d6820020 # v4.4.0
with:
node-version: '22.23.4'
- name: Install locked dependencies and the real Agents SDK
run: |
corepack enable
pnpm --filter @traceai/fi-instrumentation-livekit... install --frozen-lockfile --ignore-scripts
npm install --prefix "$RUNNER_TEMP/livekit-agents" --ignore-scripts --no-audit --no-fund \
"@livekit/agents@${{ matrix.agents }}" "zod@^3.25.76"
- name: Build
run: |
pnpm run build:deps
pnpm --filter @traceai/fi-instrumentation-livekit build
- name: Real Agents session recording with loopback-only networking
run: |
sudo unshare --net -- sh -c 'ip link set lo up; exec "$@"' sh \
env "LIVEKIT_AGENTS_DIR=$RUNNER_TEMP/livekit-agents" "$(command -v node)" \
--test packages/traceai_livekit/test/agents-recording.cjs
git diff --exit-code -- pnpm-lock.yaml
19 changes: 15 additions & 4 deletions typescript/packages/traceai_elevenlabs/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -107,9 +107,20 @@ The optional `OTEL_EXPORTER_OTLP_TRACES_ENDPOINT` and `ELEVENLABS_BASE_URL` over
| `textToSpeech.streamWithTimestamps` | You finish or stop iterating timestamp chunks |
| `speechToText.convert` | Synchronous transcription completes, including multi-channel output |

Use these methods on the wrapped client as usual. `withRawResponse()`, `then`, `catch`, `finally`, byte-stream readers (including BYOB when the upstream stream supports it), iteration and piping remain available. Timestamp responses retain audio and alignment data in your application, but that data is not exported. Requests that set `webhook: true`, realtime/WebSocket APIs and other SDK namespaces pass through without integration spans.
Use these methods on the wrapped client as usual. `withRawResponse()`, `then`, `catch`, `finally`, byte-stream readers (including BYOB when the upstream stream supports it), iteration and piping remain available. Timestamp responses retain audio and alignment data in your application. By default, neither is exported. Requests that set `webhook: true`, realtime/WebSocket APIs and other SDK namespaces pass through without integration spans.

Hosted ElevenLabs Agents conversations are a separate integration. This package does not export conversation webhooks, audio attachments, usage tokens or estimated costs.
Hosted ElevenLabs Agents conversations are a separate integration. This package does not export conversation webhooks, usage tokens or estimated costs.

## Optional binary attachments

Binary capture is off by default. If your destination supports the attachment service, explicitly enable your provider's attachment manager and pass that **same manager** as `attachments: provider.attachments` to `ElevenLabsInstrumentation`. Await `provider.attachments.ready()` at startup to avoid missing early calls; continue to use the provider's `shutdown()` for delivery. The adapter never creates another uploader or enables recording.

- `textToSpeech.convertWithTimestamps` can attach a complete MP3 response up to 256 KiB. The output format must be the SDK default or a supported MP3 format. Generated audio is not proof of playback.
- `speechToText.convert` can attach an existing `Blob`/`File` when you also set `attachmentInputMimeType` to its exact MIME type: `audio/wav`, `audio/mpeg`, `audio/ogg`, `audio/mp4`, `audio/webm` or `audio/flac`. Filenames are not exported; mismatched or missing types are not sniffed.
- `textToSpeech.convert`, `textToSpeech.stream` and `textToSpeech.streamWithTimestamps` can attach the MP3 your application **reads to the end**, up to 4 MiB per call (about 4 minutes at the default 128 kbit/s), or up to the server's single-upload limit when that is smaller. Tracing copies only the chunks you already read; it never reads ahead, tees or drains the stream. If you cancel, stop iterating, hit an error or exceed that cap, nothing is uploaded: a partial file is never stored as if complete. While a stream is being read, the manager reserves the whole cap for it (4 MiB, or the smaller server limit) out of its shared in-memory budget of about 31 MiB. With 4 MiB reservations about 7 streams can be captured at once on an otherwise idle manager; further concurrent streams report `queue_full` and pass through uncaptured.
- PCM/ulaw output, local paths and remote URLs are not captured. Alignment data stays in your SDK response, not in attachment metadata.

Attachments obey the manager's default-off, server, sampling and PII policies in addition to the instrumentation's directional hide settings. Without a qualified speech sanitizer, `piiRedaction` denies capture. Existing raw audio may contain embedded metadata; a policy requiring metadata removal also denies it. The span's `elevenlabs.attachment.input` / `elevenlabs.attachment.output` value is the admission result; `queued` means admitted for transfer, **not** validated or playable. Content is out of band, never base64 in OTLP. A call made through this bound instrumentation carries the shared media-owner context marker (`traceai.auto.attachment_owner.v1`), so wire-level capture that honours the marker does not upload the same audio again.

## Privacy

Expand All @@ -123,7 +134,7 @@ export FI_HIDE_ERROR_DETAILS=true

`FI_HIDE_INPUT_MESSAGES` / `FI_HIDE_INPUT_TEXT` and their output equivalents also hide the corresponding speech text. You can pass core `traceConfig` options to the instrumentation instead. These settings affect traces, not SDK request or response values.

The package never exports raw audio, base64, alignment text, upload paths, source URLs, request headers or provider error bodies. Errors record an error status, a fixed error type, and an HTTP status when available, not exception messages or stack traces—even if `FI_HIDE_ERROR_DETAILS` is false. Your application is responsible for content it adds to parent spans and context metadata.
Trace attributes never contain raw audio, base64, alignment text, upload paths, source URLs, request headers or provider error bodies. Binary audio leaves the process only through the optional attachment manager described above. Errors record an error status, a fixed error type, and an HTTP status when available, not exception messages or stack traces—even if `FI_HIDE_ERROR_DETAILS` is false. Your application is responsible for content it adds to parent spans and context metadata.

## Stream lifecycle

Expand All @@ -139,5 +150,5 @@ For module-wide registration, call `instrumentation.manuallyInstrument(sdk)` wit

- **No span after receiving audio:** consume or cancel the returned stream. Receiving headers is not successful completion. Use `shutdown()` before exiting a short script.
- **Spans appear separately from your application trace:** initialize an async context manager once and call the wrapped client within the active application span.
- **Text is missing:** check the `FI_HIDE_*` variables and the instrumentation's `traceConfig`. Audio and provider error text are intentionally never captured.
- **Text is missing:** check the `FI_HIDE_*` variables and the instrumentation's `traceConfig`. Binary audio needs the separate attachment opt-in; provider error text is never captured.
- **Transcription spans are missing:** `webhook: true` submits an asynchronous job; this package does not trace its later completion.
190 changes: 190 additions & 0 deletions typescript/packages/traceai_elevenlabs/src/attachments.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,190 @@
import type { Span } from "@opentelemetry/api";
import type { TraceConfig } from "@traceai/fi-core";
import type { AttachmentManager, AttachmentStream } from "@traceai/fi-core/attachments";
import { record, safely } from "./operation.js";

export type ElevenLabsInputAudioMimeType =
| "audio/wav"
| "audio/mpeg"
| "audio/ogg"
| "audio/mp4"
| "audio/webm"
| "audio/flac";

// A bounded complete-response path for convertWithTimestamps (one JSON value).
const MAX_DECODED_BYTES = 256 * 1024;
/**
* Requested consumed-stream cap: about 4.4 minutes of the SDK's default 128 kbit/s MP3. The
* manager reserves min(this, the server's single-upload limit, its owned budget) when a stream is
* admitted, so a server whose `max_upload_bytes` is below 4 MiB still admits the stream with a
* smaller cap. The reservation bounds per-call memory. A response above the effective cap uploads
* nothing (`too_large`); a truncated object is never uploaded as complete.
*/
export const ELEVENLABS_STREAM_MAX_BYTES = 4 * 1024 * 1024;
const INPUT_TYPES: readonly string[] = [
"audio/wav", "audio/mpeg", "audio/ogg", "audio/mp4", "audio/webm", "audio/flac",
];
const MP3_FORMAT = /^mp3_(22050_32|24000_48|44100_(32|64|96|128|192))$/;
const BASE64 = /^[A-Za-z0-9+/]*={0,2}$/;

/** Canonical, bounded base64 only. Buffer.from tolerates malformed input, so check first. */
function decode(encoded: unknown, maxBytes: number): Uint8Array | undefined {
if (typeof encoded !== "string" || encoded.length === 0 ||
encoded.length > Math.ceil(maxBytes / 3) * 4) return undefined;
if (encoded.length % 4 !== 0 || !BASE64.test(encoded)) return undefined;
const decoded = Buffer.from(encoded, "base64");
if (decoded.length === 0 || decoded.length > maxBytes) return undefined;
if (decoded.toString("base64") !== encoded) return undefined;
return decoded;
}

/**
* Adapters never own an uploader or change the provider manager's lifecycle. Every capture
* goes through the shared manager's admission before any read, decode or copy.
*/
export class SpeechAttachments {
private stream?: AttachmentStream;
private streamState: "idle" | "open" | "closed" = "idle";

constructor(
private readonly manager: AttachmentManager,
private readonly span: Span,
private readonly policy: TraceConfig,
private readonly outputFormat: string | undefined,
) {}

private allowed(direction: "input" | "output"): boolean {
// Adapter-local restrictions can only tighten the shared manager's policy.
// No qualified speech sanitizer is available here.
return !this.policy.piiRedaction && (direction === "input"
? !(this.policy.hideInputs || this.policy.hideInputMessages || this.policy.hideInputText)
: !(this.policy.hideOutputs || this.policy.hideOutputMessages || this.policy.hideOutputText));
}

/** Raw PCM/ulaw lack sample-format metadata in v1; never label those bytes MP3. */
private mp3(): boolean {
return this.outputFormat === undefined || MP3_FORMAT.test(this.outputFormat);
}

input(request: unknown, mimeType: ElevenLabsInputAudioMimeType): void {
safely(() => {
if (!INPUT_TYPES.includes(mimeType)) return;
const result = this.manager.attach(this.span, {
kind: "Blob",
// Core checks default-off, server policy, PII, scope and sampling BEFORE
// this getter. No file/stream property is inspected on a denied capture.
get data(): Blob {
const file = record(request).file;
if (!(file instanceof Blob) || file.type !== mimeType) {
throw new Error("unsupported_audio_source");
}
return file;
},
}, {
mimeType,
direction: "input",
purpose: "model_input",
capture: { captureContent: this.allowed("input") },
});
this.span.setAttribute("elevenlabs.attachment.input", result.admission);
});
}

timestamped(value: unknown): void {
safely(() => {
if (!this.mp3()) {
this.span.setAttribute("elevenlabs.attachment.output", "source_unavailable");
return;
}
const result = this.manager.attach(this.span, {
kind: "Bytes",
get data(): Uint8Array {
const decoded = decode(record(value).audioBase64, MAX_DECODED_BYTES);
if (!decoded) throw new Error("unsupported_audio_source");
return decoded;
},
}, {
mimeType: "audio/mpeg",
direction: "output",
purpose: "model_output",
capture: { captureContent: this.allowed("output") },
});
this.span.setAttribute("elevenlabs.attachment.output", result.admission);
});
}

/**
* Admit the consumed stream once per call, before any byte is copied or decoded. A denied
* admission retains nothing and is never retried for this call.
*/
private open(): AttachmentStream | undefined {
if (this.streamState === "idle") {
this.streamState = "closed";
if (!this.mp3()) {
this.span.setAttribute("elevenlabs.attachment.output", "source_unavailable");
return undefined;
}
const stream = this.manager.startAttachmentStream(this.span, {
mimeType: "audio/mpeg",
direction: "output",
purpose: "model_output",
maxBytes: ELEVENLABS_STREAM_MAX_BYTES,
capture: { captureContent: this.allowed("output") },
});
if (stream.reason !== "queued") {
this.span.setAttribute("elevenlabs.attachment.output", stream.reason);
return undefined;
}
this.stream = stream;
this.streamState = "open";
}
return this.streamState === "open" ? this.stream : undefined;
}

/**
* Observe one chunk the APPLICATION already pulled through the instrumentation-owned bridge
* (design 6.4). tryWrite copies synchronously, so a BYOB view may be handed back right after.
*/
chunk(bytes: Uint8Array): void {
safely(() => {
if (this.streamState === "closed" || !(bytes instanceof Uint8Array) || bytes.byteLength === 0) return;
const stream = this.open();
if (stream && !stream.tryWrite(bytes)) this.close("incomplete");
});
}

/** One base64 chunk of a consumed timestamp stream; decoded only after admission. */
timestampChunk(value: unknown): void {
safely(() => {
if (this.streamState === "closed") return;
const encoded = record(value).audioBase64;
if (typeof encoded !== "string" || encoded.length === 0) return;
const stream = this.open();
if (!stream) return;
const decoded = decode(encoded, ELEVENLABS_STREAM_MAX_BYTES);
// A malformed or oversized chunk means the object would be incomplete: upload nothing.
if (!decoded || !stream.tryWrite(decoded)) this.close("incomplete");
});
}

/**
* Called by the operation BEFORE its span ends, so a finished stream's reference is
* projected on a recording span. Anything but a complete read aborts: nothing uploads.
*/
close(completion: "complete" | "incomplete"): void {
safely(() => {
const stream = this.stream;
const wasOpen = this.streamState === "open";
this.stream = undefined;
this.streamState = "closed";
if (!stream || !wasOpen) return;
if (completion === "complete") {
const handle = stream.finish();
this.span.setAttribute("elevenlabs.attachment.output", handle.admission);
} else {
stream.abort("incomplete");
this.span.setAttribute("elevenlabs.attachment.output", stream.reason);
}
});
}
}
1 change: 1 addition & 0 deletions typescript/packages/traceai_elevenlabs/src/index.ts
Original file line number Diff line number Diff line change
@@ -1,3 +1,4 @@
export { ElevenLabsInstrumentation } from "./instrumentation.js";
export type { ElevenLabsInstrumentationConfig } from "./instrumentation.js";
export { VERSION } from "./version.js";
export type { ElevenLabsInputAudioMimeType } from "./attachments.js";
26 changes: 23 additions & 3 deletions typescript/packages/traceai_elevenlabs/src/instrumentation.ts
Original file line number Diff line number Diff line change
@@ -1,6 +1,7 @@
import {
Attributes,
context,
createContextKey,
SpanKind,
trace,
TracerProvider,
Expand Down Expand Up @@ -28,6 +29,11 @@ import {
safely,
TEXT_LIMIT,
} from "./operation.js";
import type { AttachmentManager } from "@traceai/fi-core/attachments";
import { SpeechAttachments, ElevenLabsInputAudioMimeType } from "./attachments.js";

/** Context marker read by `@traceai/auto` wire capture; `createContextKey` is `Symbol.for`. */
const ATTACHMENT_OWNER = createContextKey("traceai.auto.attachment_owner.v1");
import { observeResponse } from "./responses.js";
import { VERSION } from "./version.js";

Expand Down Expand Up @@ -68,6 +74,10 @@ export interface ElevenLabsInstrumentationConfig {
instrumentationConfig?: InstrumentationConfig;
tracerProvider?: TracerProvider;
traceConfig?: TraceConfigOptions;
/** The configured provider's shared manager. Omitted means no binary capture. */
attachments?: AttachmentManager;
/** Explicit MIME for STT Blob/File inputs; no sniffing or stream reads. */
attachmentInputMimeType?: ElevenLabsInputAudioMimeType;
/** Absolute span lifetime, including request and consumption. Default 300000 ms. */
maxSpanDurationMs?: number;
/** Bound active telemetry, not SDK calls. Excess calls pass through. Default 1024. */
Expand Down Expand Up @@ -272,9 +282,8 @@ export class ElevenLabsInstrumentation extends InstrumentationBase {
if (voice) attributes["elevenlabs.voice.id"] = voice;
// Unlike JSON body overrides, the SDK skips nullish query values.
const query = stt ? {} : record(options.queryParams);
const format = identifier(
bodyField(query, "output_format", undefined) ?? request.outputFormat,
);
const formatValue = bodyField(query, "output_format", undefined) ?? request.outputFormat;
const format = identifier(formatValue);
if (format) attributes["elevenlabs.output_format"] = format;
if (
!stt &&
Expand All @@ -295,15 +304,26 @@ export class ElevenLabsInstrumentation extends InstrumentationBase {
{ kind: SpanKind.CLIENT, attributes },
callContext,
);
const attachments = this.options.attachments
? new SpeechAttachments(this.options.attachments, span, policy,
formatValue === undefined ? undefined : format ?? "unsupported")
: undefined;
operation = new Operation(
span,
policy,
bounded(this.options.maxSpanDurationMs, 300_000, 86_400_000),
(ended) => this.active.delete(ended),
attachments,
);
if (stt && this.options.attachmentInputMimeType &&
!(options.abortSignal instanceof AbortSignal && options.abortSignal.aborted))
attachments?.input(request, this.options.attachmentInputMimeType);
this.active.add(operation);
operation.watchAbort(record(args[stt ? 1 : 2]).abortSignal);
callContext = trace.setSpan(callContext, span);
// This adapter owns the call's media, so wire-level auto capture must not upload the
// same bytes again (shared key; no dependency on the auto package).
if (attachments) callContext = callContext.setValue(ATTACHMENT_OWNER, true);
}
} catch {
diagnose();
Expand Down
Loading
Loading