Telemetry by pblazej · Pull Request #1108 · livekit/client-sdk-swift

pblazej · 2026-09-03T11:26:28Z

Telemetry

Draft: integrates the shared Rust telemetry core (livekit-telemetry, livekit/rust-sdks#1396) into the Swift SDK, per the "Client Telemetry" design doc. The core owns policy and vocabulary; the SDK contributes a transport, instruments and the wiring. The Android (livekit/client-sdk-android) and Flutter (livekit/client-sdk-flutter) drafts follow the same shape.

Design

Telemetry is the one entry point: LiveKitSDK.setTelemetry(options) / Telemetry.configure(options) configure the process-wide pipeline (it lives in the Rust core) at any time; every Room takes a scope with its own trace id at init (room.telemetryTraceId), so pre-connect failures are captured and nothing leaks a session handle. TelemetryOptions mirrors the core's config; nothing from UniFFI is public.
Instruments, selectable with TelemetryOptions.instruments (.room, .rtc, .logs, .device; all by default): DeviceTelemetry (thermal, low power, memory, network, battery, app state, audio route and interruptions), warn/error records from the SDK, the Rust core and WebRTC through LogHub, and per Room the lk.connect / lk.reconnect / lk.publish spans with their checkpoints; RTCTelemetry reports the remote tracks' lifecycle and the core runs the lk.subscribe span itself; each track's stats timer hands the raw getStats() report to the core, which maps and windows it.
Span is the core's span with Swift helpers and a task-local Span.current; Tracing / setTracing remain as a deprecated no-op.
Telemetry.setAttribute attaches pipeline-wide attributes; the transport is a dedicated URLSession with networkServiceType = .background that returns the collector's raw answer (the core decides retry / drop / go-silent); the destination is derived from the connect URL unless TelemetryOptions.endpoint is set.

Testing

TelemetryTests runs end to end against livekit-server --dev and an otelcol-contrib writing OTLP/JSON to disk (Tests/LiveKitCoreTests/Telemetry/otelcol.yaml); CI downloads and runs the collector next to the dev server. The headless macOS harness in the same suite drives connect, publish, first media, quick and full reconnect and disconnect on synthetic media; otelcol-lgtm.yaml fans the same stream into a local Grafana LGTM (make telemetry-harness). Package.swift points at a locally built UniFFI package for now.

Integrates livekit-telemetry (livekit-uniffi, local debug package for now) as a connection-scoped subsystem, with the bare minimum end to end: error reporting and RTC stats reporting. - RoomOptions.telemetry: TelemetryOptions (endpoint, headers, storage directory defaulting to Caches/livekit-telemetry, flush interval, stats window). Prototype surface: the design routes enablement through the token's observability grant. - RoomTelemetry on ConnectionDependencies: created when connect() starts, torn down with the connection. Registers on every published/subscribed track (MetricsManager's pattern), enables reportStatistics, maps TrackStatistics to RtcStatsSample per RTP stream; the core windows them into lk.rtc.stats.sample. Sets lk.room.sid/name and lk.participant.sid/identity as session attributes once connected. - Logger: an internal LogRelay next to Loggable.log fans out .warning and .error calls as structured LogRecords to in-process sinks; the app-facing Logger protocol and sharedLogger are untouched, debug/info never take this path. RoomTelemetry is a sink and emits them as OTLP log records with code.function / code.file.path / code.line.number / lk.log.type. - Device state: ProcessInfo thermal + power-state notifications and AppStateListener feed Telemetry.setDeviceState (events + cadence). - URLSessionTelemetryTransport: the host's dumb bytes mover, mapping HTTP outcomes onto ExportError. - E2E test (needs livekit-server --dev and grafana/otel-lgtm): a room publishes synthetic video, logs an error, and the test asserts the outbound video window, the session attributes and the error record in Loki. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

The core constructor now takes an optional transport (falling back to the HTTP client registered with livekit-net) and can fail when neither exists. RoomTelemetry.init is failable and ConnectionDependencies skips telemetry rather than failing the connection. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

The locally built LiveKitUniFFI now carries the data-streams-v2 types, which introduce a second EncryptionType visible to the test support target. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

Span is no longer a stopwatch: kind, outcome (ok | error | cancelled), typed attributes, an error type, and a SpanContext (trace id + span id) assigned by the telemetry sink. record() keeps its shape - the nine connect checkpoints are span events now. Fan-out instead of replacing LoggingTracer: RoomTracer asks the app-facing Tracing (LiveKitSDK.setTracing, slot untouched) for the Span, then the Room's sinks observe it - RoomTelemetry (beginSpan/addSpanEvent/endSpan on the Rust core, under the session trace) and, in debug builds, an os_signpost sink (Instruments' Points of Interest, as opentelemetry-swift does). Room gets a `tracer`; ConnectionDependencies owns it. Spans wired: lk.connect (kind client, lk.connect.attempt, ends ok / error / cancelled from the connect catch) and lk.reconnect (lk.reconnect.reason, mode and attempts as attributes, one "attempt <n> <mode>" checkpoint per attempt, cancelled when disconnect() or a newer cycle wins). Warn/error records emitted during connect point at the connect span. E2E: the telemetry test now also asserts a trace rooted at lk.connect in Tempo (Grafana datasource proxy). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

@TaskLocal

- Span.current (@TaskLocal): bound around the connect attempt and the reconnect cycle. RoomTracer.beginSpan defaults its parent to it, so spans nest without handles being passed around; warn/error records emitted in a task with an ambient span point at it (the connectSpan heuristic is gone). - lk.publish: LocalParticipant._publish wraps the publish sequence in a span (lk.track.kind/source, sid on success; ok / error / cancelled), nested under the connect span for the pre-connect microphone. - lk.subscribe: from the intent to subscribe (remote publish under autoSubscribe, or the manual subscribe) to the first media - the first stats sample with bytes received, the natural end of "time to media"; "subscribed" checkpoint on the way; cancelled on unsubscribe/unpublish, error on subscription failure or after 30 s without media. - E2E: the subscriber room carries telemetry too; the session trace of each room must contain lk.connect and, respectively, lk.publish or lk.subscribe (one trace per session holds several root spans, so span names are checked, not Tempo's single root name). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

RoomTelemetry now observes memory pressure (DispatchSource), the network path (NWPathMonitor: type, expensive, constrained), battery level and charging (iOS/visionOS) and the audio session (route changes and interruptions as events), alongside the existing thermal, low-power and app-state hooks. Everything is notification-driven; nothing polls or samples CPU. The core turns these into cadence stretching and upload holds. The transport moves to a dedicated ephemeral URLSession with networkServiceType = .background: the local stack queues telemetry behind best-effort media and signaling (fq_codel BK class, Wi-Fi AC_BK) and its TCP flows switch to LEDBAT while foreground traffic is active. One connection per host, 10 s request timeout. The e2e test additionally asserts the initial thermal, memory-pressure and network-path values reach the collector. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

Apps can record their own events in the session trace next to the SDK's: `room.emitTelemetryEvent("checkout.started", attributes: ["acme.step": .int(3)])`. The core namespaces them under `custom.` and applies the same flood guard; a no-op when telemetry is off. The e2e test emits one and asserts it reaches the collector with its attribute. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

…ruments Telemetry now starts with the SDK, not with connect(): `LiveKitSDK.setTelemetry(options)` configures a process-wide `TelemetryHub` (core pipeline, transport, destination, log relay sink), so pre-connect work - track creation, permission failures, connect attempts that never reach a server - is captured. `RoomOptions.telemetry` is gone. Each Room gets a session (own trace id) from the hub in `init`, exposed as `room.telemetryTraceId`; `emitTelemetryEvent` works before connect. The destination is derived from the first connect (`https://<host>/observability/logs/otlp/v0`, `Authorization: Bearer <token>`) unless the options name an endpoint; until then everything is buffered on device. The instruments are three independent components, one per area of the design doc: `DeviceTelemetry` (process-level: thermal, low power, memory, network, battery, app state, audio session), `RoomTelemetry` (session identity, spans, subscribe spans, app-defined events) and `RTCTelemetry` (track statistics -> session; one signal out, `onFirstMedia`, which the Room wires to the subscribe spans). `ConnectionDependencies` only settles open spans and flushes when a connection ends. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

The shape every client SDK converges on (Embrace CaptureService, Sentry integrations, OpenTelemetry Instrumentation / AndroidInstrumentation): an instrument is a platform-side object with an explicit lifecycle, owned and listed by a core. DeviceTelemetry, RoomTelemetry and RTCTelemetry conform; the hub starts and stops the process-level instruments, the Room its own (started in init, stopped in deinit, which also settles the session's open spans). Kept on the Swift side rather than as a UniFFI foreign trait: the design doc's FFI is push-only, and uniffi-dart cannot call foreign traits from Rust threads. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

…r spans Review round on the Swift surface: - `TelemetryHub` becomes `Telemetry`, a global actor. Options are set statically before use (`LiveKitSDK.setTelemetry` / `disableTelemetry`, like the logger); the core pipeline is built once on first use and latched. Swift-side instrument state is actor-isolated; the thread-safe core handles stay nonisolated so Room.init and span hooks need no hop. - `LoggingTelemetry` is its own instrument. The log relay runs before the app's `Logger`, so `setLogger` cannot displace telemetry, and carries every level; the instrument decides what leaves the device. - Spans: `SpanSink`, the per-span sink arrays and `SignpostSpanSink` are gone. A `Span` has `onEnd` (public, as before) and an internal `onRecord`; `RoomTracer` hands new spans to `RoomTelemetry.attach`, which gives them identity and wires both hooks. - `lk.subscribe` moves to `RTCTelemetry`: its end is an RTC fact (first inbound bytes), so no first-media signal between instruments. - `Telemetry.setAttribute` sets pipeline-wide attributes (every record of every session). - The e2e test reads OTLP/JSON written by `otelcol-contrib` (file exporter, `Tests/LiveKitCoreTests/Telemetry/otelcol.yaml`) instead of querying Loki and Tempo; CI downloads and runs the collector next to the dev server. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

…racer One tracer. As the default `Tracing` it creates spans and logs them at debug when they end, as `LoggingTracer` did (kept as a deprecated alias). Bound to a Room's telemetry session it also gives each span an identity in the session's trace at creation; the `Span` then reports checkpoints and its end to the session itself, so the post-hoc `attach` and its closures are gone. An app-injected `Tracing` still creates the span objects. `kind` is a required argument on the Room-scoped `beginSpan`, so a one-argument call cannot resolve to the protocol overload and skip the binding - which is how `lk.publish` briefly went missing. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

RoomTelemetry had become an actor-isolated class with no isolated state: every member forwarded to the thread-safe session handle and had to be `nonisolated`. The Room now holds `telemetrySession` directly (identity attributes on connect, `emitTelemetryEvent`, `telemetryTraceId`), the tracer takes the session, and the connection's teardown flushes. Room-scoped instruments are `RTCTelemetry` and `TelemetryTracer`. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

`Telemetry.shared.configure(options)` may be called at any time: before the pipeline exists it shapes it, afterwards the destination and headers apply at once, `nil` stops it after a bounded flush - no latching, no replacing. The pipeline bootstraps on the first `register(room)` from `Room.init`, which also starts the process-level instruments and creates the Room's session and `RTCTelemetry` in a registry keyed by the Room. `Room` holds no telemetry state any more. Its whole internal surface is `register` / `unregister` / `roomDidConnect` / `connecting` / `traceId(for:)` / `emit(_:from:)` / `tracer(for:)`; `TelemetrySession` never leaves the telemetry files. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

Options, core, attributes, rooms and instruments are plain actor state; the StateSync is gone. Entry points are async: fire-and-forget where nothing comes back (register in init, unregister in deinit, emit, configure), awaited where a value does (tracer(for:) in connect(), telemetryTraceId as a get-async property). Sessions are created lazily on first use, so call ordering cannot matter, and state is updated before every await so re-entrancy cannot duplicate a session or the pipeline. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

`Loggable.log` hands warn/error records to `Telemetry.log` directly: timestamp and ambient span are captured where the log happened, delivery hops to the actor. `LogRelay`, `LogSink`, `LogRecord` and `LoggingTelemetry` are gone. Rust-core logs have a single consumer, so `FFILogForwarder` drains the stream once and fans out - to the console handler `OSLogger` registers and, at warn/error, to telemetry - started by whichever needs it first at the lowest level asked for. WebRTC gets its own callback-logger sink at warning level. Records carry `lk.log.source = sdk | ffi | webrtc`. `TelemetryOptions.instruments` (`.room`, `.rtc`, `.logs`, `.device`; default `.all`) decides what the pipeline starts; app-defined events and session identity are always on. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

Logs from outside the SDK's own logger go through a single generic `LogForwarder`: `LogForwarders.ffi` drains the core's single-consumer log stream, `LogForwarders.rtc` owns the one WebRTC callback logger. Each is started by whoever needs it first (`OSLogger`, or telemetry when the app uses its own logger) at the lowest level asked for, and fans out to the console handlers that registered and, at warn/error, to `Telemetry.log`. `Telemetry` no longer owns a WebRTC sink of its own. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

Every log line the SDK produces or captures - its own `Loggable.log` calls, the Rust core's stream, WebRTC's callback logger - enters `LogHub` once as a `LogRecord` value captured where it happened (level, origin, code location, timestamp, ambient span) and fans out two ways: to the app's `Logger`, which filters by its own level and receives the span as `lk.trace_id` / `lk.span_id` metadata, and, at warn/error, to telemetry. Neither side learns the other's level (swift-log's MultiplexLogHandler + MetadataProvider, swift-otel's OTelLogRecord, Datadog's console vs remote threshold). External sources (`LogSources.ffi`, `.rtc`) are subscribed once per process at the lowest level any consumer asked for; `OSLogger`'s `rtc:`/`ffi:` flags request them for the console, telemetry requests warn/error. A custom `Logger` now receives WebRTC and Rust-core records too, typed `WebRTCLog` / `FFILog`. The core's own warnings are kept out of telemetry to avoid a rejected-batch feedback loop. No public logger API changed. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

The iOS-only battery reader carried both attributes, which the compiler rejects; macOS builds never compiled the block. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

…ixes From the first run on an iPhone: - `TelemetryOptions.logLevel` (default `.warning`) is the threshold a log record needs to leave the device; WebRTC's own logs go from `.error` regardless - 60 of 77 records in one call were WebRTC's internal warnings (duplicate codecs, disabled field trials, RTCP timeouts). - Device state no longer seeds the network path from `currentPath`, which reads `unavailable` on iOS before the monitor starts; the monitor's first update carries the first device state. - Audio route and interruption events carry a display body. - The e2e test enforces the rule: every record has a body, log records are warn/error only, events carry `otel.event.name`. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

An error logged inside a span bound to room 0 must carry that span's id and land in that Room's trace; a record logged outside any span lands in the process session. Caught a core bug: records arriving after their span ended were misfiled. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>