Skip to content

feat(python): Pipecat recorder, LiveKit owned-file and Egress import on the shared attachment core - #678

Draft
nik13 wants to merge 10 commits into
feat/attachments-python-corefrom
feat/attachments-python-voice
Draft

nik13 wants to merge 10 commits into
feat/attachments-python-corefrom
feat/attachments-python-voice

Conversation

@nik13

@nik13 nik13 commented Nov 11, 2026 •

Copy link
Copy Markdown
Contributor

Part of TH-2379 (Linear).

Summary

Draft, head 1d02a461 (review follow-ups on 77f18b26, see "Review follow-ups"; earlier follow-ups on 74eb15b1: Pipecat per-turn audio, short-call binding, LiveKit Voice root docs). Stacked on #676 (feat/attachments-python-core). This merges core head 5a7e1105 and moves every voice adapter onto the shared core API: preflight, start_recording, segments plus manifests, attach_owned_recording and import_completed_recording. Attachments stay off unless enabled. Creating a new recording is separately off.

V01: Pipecat recorder (traceai_pipecat.attachments)

  • bind_audio_processor(processor, AudioSessionContext.from_pipeline(worker), manager=..., layout="auto"|"separate_tracks") observes the app's existing AudioBufferProcessor.

    • It listens to on_audio_data, or to on_track_audio_data for separate_tracks, and to on_recording_stopped.
    • Each chunk goes straight into a core RecordingWriter, which writes WAV segments of up to 30 s and publishes a manifest with gaps.
    • It never starts, stops or reconfigures the recorder, never reads Pipecat's internal buffers, and never falls back to an ambient span.
  • Bound to the real conversation. The recording attaches to the conversation span that Pipecat's own TurnTraceObserver started; turn spans are its children. The writer opens on the first chunk, so a conversation span that starts asynchronously is still found.

    • The upload carries Pipecat's conversation.id as conversation_id.
    • Without Pipecat tracing, the status is correlation_unavailable and nothing is kept.
  • Layouts:

    Processor or option Layout Tracks
    mono mixed one mixed track
    stereo stereo user = left = input, bot = right = output
    separate_tracks separate_tracks one track per side

    The core evaluates mixed and stereo as mixed, so any hide flag denies the whole recording. With separate_tracks, a hidden side becomes a policy_denied gap for that track and the other side still uploads.

  • Each start/stop cycle is its own recording. on_recording_stopped finishes the cycle in the background. await binding.finish() awaits uploads within the deadline (at most 30 s).

  • Bounded.

    • Upstream: 0 < buffer_size <= 128 KiB is required. buffer_size=0 returns unbounded_source; so does enable_turn_audio unless per-turn uploads are opted in with explicit caps (below).
    • Uploader: when it falls behind, overflow becomes a queue_full gap rather than memory growth (window of 2 segments).
  • create_audio_capture(...) returns a configured, unstarted processor only when both local and server record_audio allow it (operation="create_recording").

  • streaming_unavailable is retired. Support set is exactly pipecat-ai==1.15.0. 1.0.0 returns unsupported_version (checked on the VM).

Follow-ups: per-turn audio, short calls, LiveKit Voice root

  • Per-turn audio (opt-in, off by default). bind_audio_processor(..., turn_audio=TurnAudioCaps(max_turns=, max_bytes=, max_duration_ms=)) (also create_audio_capture and traceai.bind_pipecat_audio_processor). All three caps are required, per call, with ceilings 1000 turns / 64 MiB / 2 h.
    • on_user_turn_audio uploads as input, on_bot_turn_audio as output: mono WAV, purpose turn_recording, turn_id = Pipecat turn number, the call's conversation_id.
    • Bound to that turn's own span via TurnTraceObserver.get_turn_context(turn_number) (public API). Never the conversation span or an ambient span; an explicit-span context has no turn spans, so turns end correlation_unavailable.
    • A turn upload never projects fi.attachment.refs (or any fi.attachment.* attribute) onto the turn span, even while it is open: the preflight target is a CorrelationHandle. Ingest binds it on association.parent, which names the turn span.
    • Order per turn: caps on the length (no copy) -> turn span -> core preflight (deny-wins, per side) -> only then the WAV copy -> attach. An over-cap turn is skipped whole: queue_full (count) or too_large (bytes/duration). Never truncated.
    • Bind-time check for turn_recording on input/output: if both sides are denied, no turn handler is registered (turn_status = policy_denied). enable_turn_audio=True without caps stays unbounded_source; caps with a processor lacking enable_turn_audio raise ValueError.
    • binding.turn_status, binding.turn_denials (counts by reason), binding.turn_handles; await binding.finish() also waits for turn uploads within the same deadline.
    • Pipecat keeps each turn in its own memory while turn audio is on; the caps bound what the adapter copies and uploads.
  • Short calls (< buffer_size) no longer come out unsampled/correlation_unavailable. Pipecat runs on_audio_data as a task, so a short call's only chunk arrives on stop and can run after the worker's cleanup ended and cleared the conversation span. The adapter now snapshots the conversation's CorrelationHandle while the span is live (at bind, on on_recording_started, on every chunk) and binds to it when the span is gone. No deferred target was added to the core (a snapshot taken from a live, sampled span is the documented CorrelationHandle use). On the snapshot path the conversation span has no carrier stamp; ingest binds on the authenticated association.parent plus the stored span, so refs are not needed. The root marker is set at snapshot time (live, sampled, parentless span only).
  • LiveKit: how Voice finds the call (no fake root; corrected in 1d02a461 from a real worker run, see below). The SDK marks agent_session only when it is already the trace root. With tracing set up inside the entrypoint (the README Quick Start), it is: the binding marks it CONVERSATION. With tracing set up before the job, job_entrypoint (with lk.job_id) is the root and agent_session its child, so nothing is marked and ingest maps the untyped, parentless job_entrypoint when attachments are enabled on the server. A test pins the child shape (neither span gets fi.span.kind/gen_ai.span.kind).

V02: LiveKit session recording (traceai_livekit.attachments)

  • The binding no longer touches private manager internals (manager._session, manager.policy.evaluate, the status property, scope._fd).
  • Native LiveKit redaction and PII restrictions remain deny-wins.

V03: completed LiveKit Egress (traceai_livekit.recording)

  • New LiveKitEgressResolver (design 5.4). import_completed_egress(...) is now async and goes through the core's import_completed_recording.
  • Accepted only: EGRESS_COMPLETE, an audio-only request (room composite with audio_only, or a track composite with no video), and exactly one file result with known size and duration.
  • Resolver output: MIME from the container, declared_duration_ms = ceil(ns / 1e6), direction mixed, and registered_output_id from the injected EgressOutputRegistry (CQ-4).
  • Where EgressInfo comes from: an object the app already holds, or ONE fixed-ID list_egress(ListEgressRequest(egress_id=...)) through the app's client.
  • What it never does: start or stop Egress, poll, or let a location, filename, URL or credential leave the resolver.

Conversation root marker

fi.span.kind=CONVERSATION (plus gen_ai.span.kind, so the Pipecat mapper's CHAIN default doesn't contradict it) is set only on a live, sampled SDK span with no parent (local or remote), and only when the binding is admitted. For Pipecat it is set when the conversation identity is snapshotted (at bind if the span already exists, on recording start, or on a chunk), while the span is still live.

  • Pipecat's conversation span is marked when it is a true root.
  • LiveKit: which span is the root depends on where tracing is set up, checked with a real 1.8.10 and 1.8.11 worker (below). Inside the entrypoint, agent_session is the parentless root and the binding marks it. Before the job, agent_session is a child of job_entrypoint and is not marked.

Auto module (traceai)

  • bind_pipecat_audio_processor(processor, context, **options) passes options through.
  • New async import_livekit_completed_egress(...) uses the provider's shared manager.
  • bind_pipecat_audio_processor(..., turn_audio=traceai_pipecat.TurnAudioCaps(...)) passes the per-turn opt-in through.
  • attachment_status()["adapters"] now reports the binding's own status, so a denied binding is no longer reported as session_bound.

Review follow-ups on 1d02a461 (two commits on 77f18b26)

  • create_audio_capture(..., turn_audio=...) checks the turn policy first. The turn_recording probe (operation create_recording, input and output) runs BEFORE the AudioBufferProcessor is built. enable_turn_audio=True is set only when at least one side is admitted; otherwise Pipecat keeps no turn audio, no turn handler is registered and turn_status is the reason (for example policy_denied).

  • New regression tests (the review's surviving mutations):

    • a local CapturePolicy(record_inputs=False) with separate_tracks + turn_audio uploads only the output turn and records policy_denied for the input turn (no env flag involved);
    • a created recorder with turn_audio: turn 1 uploads both sides, then the server's record_audio is turned off and turn 2 is denied on both sides (recording_disabled), because turns preflight as create_recording;
    • turns denied on both sides at create time: recorder built without enable_turn_audio, turn_status=policy_denied, no turn handlers;
    • a turn uploaded while its turn span is still open: no fi.attachment.* attribute on the turn span; association.parent is the turn span.
  • Real LiveKit worker (livekit-agents 1.8.11 and 1.8.10, loopback livekit-server --dev, explicit agent dispatch, AgentSession(...).start(record=True), bind_session(..., owned_recording_files=True) against a loopback TLS F01 server, spans exported per span end):

    Tracing set up Roots exported agent_session Binding
    inside the entrypoint 4 traces: agent_session (session spans), job_shutdown (+ room_disconnect, session_end_upload, shutdown_callback), room_connect, event_loop_blocked; job_entrypoint not recorded parentless root, carries lk.job_id, fi.span.kind=CONVERSATION session_bound, root_marked=True, Ogg uploaded (validating)
    before the job (setup_fnc) 1 trace rooted at job_entrypoint (lk.job_id); agent_session and job_shutdown are its children child, not marked session_bound, root_marked=False, Ogg uploaded (validating)

    Identical on 1.8.10 and 1.8.11. The README said the inside-entrypoint trace was rooted at job_shutdown and the call could not be listed; both are corrected.

Check (private GCP VM, deleted) Result on 1d02a461
voice-tests Pipecat 1.15.0, Python 3.11 / 3.12 42 passed each; 3 repeat runs 42/42; the 4 new tests 25x on each Python, 0 failures
voice-tests LiveKit 1.8.10 (3.11) / 1.8.11 (3.12, 3.10) 51 passed each; LiveKit regressions 572 passed x3
core python/tests/attachments + python/traceai/tests 437 passed
control: head tests against the 77f18b26 adapter 1 fails: the create-time probe test (the other three pin behaviour 77f18b26 already had, now with tests that discriminate)
mutations 3 of 3 detected: turn preflight drops capture_policy (local-policy test), drops operation (record_audio revocation test), recorder built with enable_turn_audio before the probe (create-time test)
OFF probe (disabled config / DisabledAttachmentManager, poisoned args, turn_audio) disabled, nothing touched, 0 owned bytes
ruff check / format on changed files identical findings on base and head (pre-existing)

On the first VM run the revocation test failed intermittently on Python 3.11 (4 of 5 runs; 3.12 passed): revoking record_audio also stopped turn 1's still-queued output upload (deny-wins on queued work, correct behaviour). The test now waits for turn 1's uploads before revoking.

Verification of the follow-ups on 77f18b26 (private GCP VM, deleted; synthetic audio, core F01 transport double)

Suite Result
voice-tests Pipecat 1.15.0, Python 3.11 / 3.12 38 passed each (24 before + 14 new); 5 repeat runs on 3.11 all 38/38
voice-tests LiveKit 1.8.10 (3.11) / 1.8.11 (3.12, 3.10) 51 passed each; LiveKit regressions 572 passed x3
core python/tests/attachments + python/traceai/tests 437 passed (3.11)
control: these tests against the 74eb15b1 adapter (plus a TurnAudioCaps stub so the module imports) 12 fail: the forced short-call test and every per-turn test. The real-pipeline short-call test passes there too: on a quiet runner the stop chunk usually runs before cleanup, which is why the forced test pins the race.
mutations (one each, Pipecat suite) 12 of 12 detected: no snapshot fallback; no root mark at snapshot; no on_recording_started snapshot; count / bytes / duration cap removed; turn falls back to the conversation span; wrong purpose; sides swapped; turn handlers registered when both sides are denied; enable_turn_audio accepted without caps; finish() ignores turn uploads
OFF probe (disabled config and DisabledAttachmentManager, poisoned arguments, with and without turn_audio) disabled, nothing touched, 0 owned bytes
ruff F-class on changed files same 3 pre-existing findings on base and head (none in new code)

CI on 77f18b26: 13 of 13 green (8 voice component jobs incl. Pipecat 1.15.0 on 3.11/3.12, plus 5 attachment jobs on Python 3.10 to 3.14).

Verification (private GCP VM, synthetic audio, core F01 transport double + loopback TLS server)

Exact pins, one venv each (freezes in evidence):

Suite Result
voice-tests Pipecat 1.15.0, Python 3.11 / 3.12 24 passed each
voice-tests LiveKit Agents 1.8.10 (3.11) / 1.8.11 (3.11, 3.10) 51 passed each (3 envs)
existing LiveKit regressions, 1.8.10 / 1.8.11 / 1.8.11 on 3.10 572 passed each
core python/tests/attachments + python/traceai/tests (merge is clean) 437 passed (Python 3.11)
control: old voice adapters (merge 5154dee7) + the new tests LiveKit: 50 of 51 fail. Pipecat: collection error, because the old adapter has no recorder API and returned streaming_unavailable.
Pipecat 1.0.0 with the real version gate unsupported_version

Real-pipeline proof (V01). A real PipelineWorker(enable_tracing=True, enable_turn_tracking=True) runs on Pipecat 1.15.0 with synthetic tone frames and auto_start_recording. For mono, stereo and separate_tracks:

  • the uploaded segment PCM is byte-identical to what Pipecat emitted (per track for separate_tracks);
  • the manifest layout, tracks, duration_ms and capture_complete are correct;
  • the UploadRequest parent is the conversation span Pipecat started;
  • that span carries fi.span.kind=CONVERSATION and the attachment carrier stamp, and no other span does.

With an app span as parent, the conversation span is not marked.

Mutations (one line each, run against the head's tests): 9 of 9 detected:

  • Pipecat: root marker on a child span; ambient-span fallback; separate_tracks probed as mixed; unbounded upstream accepted; no finish on stop.
  • LiveKit: root marker on a child span; hard link allowed; EGRESS_ACTIVE treated as ready; video Egress accepted.

Logs and freezes are under evidence/c2-py-voice/ in the coordinator workspace (not in this repository). The VM was deleted.

CI on 74eb15b1: 13 of 13 green. That is 8 voice component jobs plus 5 attachment jobs.

  • On 878db07f, every voice job timed out in apt-get install ffmpeg because the runners could not resolve the Ubuntu mirror (DNS), on both attempts.
  • 74eb15b1 encodes the synthetic Ogg/Opus fixture with PyAV's bundled codecs and drops the apt step.
  • Re-proved on a clean VM with no system ffmpeg or libsndfile: Pipecat 24/24 (3.11, 3.12); LiveKit 51/51 (1.8.10 on 3.11, 1.8.11 on 3.12 and 3.10); LiveKit regressions 572 x3; pip check clean.

Not done / remains open (all fail closed)

  • Q02 product gate. There is no real SFU/participant call and no playback in the product UI yet. The backend import route currently refuses (403), and B08 READERS is empty, so V03 is proven against the core's synthetic submit_import only.
  • CQ-4. The EgressOutputRegistry implementation (how an app learns registered_output_id) is backend/contract work. Without a registry, the status is source_unavailable.
  • LiveKit Voice listing limits. With tracing inside the entrypoint, a call is listed only through an admitted recording binding (which marks agent_session); without one, nothing marks a root. With tracing before the job, listing relies on the server's job_entrypoint mapping, which the collector also needs the attachments flag for. Traces ingested before the server flag are not re-classified. Listing and playback in the product are not proven end to end here (Q02).
  • LiveKit single-file recording state. attach_owned_recording tier single sends no recording_id, so the server's recording projection has no manifest and shows it as pending. Either the SDK emits a one-segment manifest for single files or the server synthesizes one; this needs a cross-lane decision and is not changed here.
  • Per-turn audio on a real call. Proven with a real Pipecat pipeline and Pipecat's own turn events on synthetic frames, not a real SFU call; the bot side of a real pipeline is covered by Pipecat-dispatched turn events (queued synthetic system frames overtake bot audio).
  • Pipecat support range. Versions 1.1 to 1.14 are not qualified (unsupported_version).
  • Core floor. The framework packages' fi-instrumentation-otel floor must be raised in the release PR that first publishes a core with attachments (the same caveat as feat(python): attachment core with bounded lifecycle, preflight, recordings and imports #676).

E2E coverage

not applicable (no product e2e/ harness in this repository) · flows hit: n/a · marker: n/a · verdict: not applicable. Product voice playback is the separate Q02 / OBS-E2E-061 gate, not a skipped pass.

🤖 Generated with Claude Code

nik13 and others added 5 commits November 10, 2026 18:57
Keep Pipecat streams and completed Egress imports fail-closed until the shared transport prerequisites are qualified.

Co-Authored-By: Claude Code <noreply@anthropic.com>
Co-Authored-By: Claude Code <noreply@anthropic.com>
… import on the shared core

V01: bind_audio_processor observes the app's AudioBufferProcessor
(on_audio_data / on_track_audio_data) into a core RecordingWriter bound to
the conversation span Pipecat's own TurnTraceObserver started; mixed, stereo
and separate_tracks manifests; recording stays off unless enabled;
create_audio_capture needs local and server record_audio.

V02: LiveKit session binding uses public preflight, identify() and
attach_owned_recording (tier single, direction mixed); no private manager
or scope internals.

V03: LiveKitEgressResolver + async import_completed_egress through the
core import_completed_recording seam (completed audio-only Egress only).

fi.span.kind=CONVERSATION is set only on a live, parentless SDK span.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…t-in voice recordings

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@nik13 nik13 changed the title feat(python): add opt-in voice attachment binding components feat(python): Pipecat recorder, LiveKit owned-file and Egress import on the shared attachment core Nov 11, 2026
The runner apt mirror failed DNS on both attempts. The synthetic Ogg/Opus
fixture is now encoded with PyAV's bundled codecs (already a LiveKit
Agents dependency), so the apt step is removed.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
nik13 and others added 4 commits November 11, 2026 18:59
…ce root docs

- Pipecat per-turn uploads (opt-in): `turn_audio=TurnAudioCaps(max_turns=,
  max_bytes=, max_duration_ms=)` on bind_audio_processor/create_audio_capture.
  on_user_turn_audio -> input, on_bot_turn_audio -> output, purpose
  turn_recording, bound to Pipecat's own turn span via
  TurnTraceObserver.get_turn_context(turn_number). Caps are per call, checked
  on length before any copy; an over-cap turn is skipped whole (queue_full for
  the count, too_large for bytes/duration). enable_turn_audio without caps is
  still refused as unbounded_source. No fallback to the conversation or an
  ambient span. Turn handlers are not registered when both sides are denied.
- Short calls: the conversation identity is snapshotted while Pipecat's
  conversation span is live (bind, on_recording_started, every chunk), so a
  call shorter than buffer_size whose only chunk runs after the worker ended
  the span still binds to that conversation instead of coming out
  unsampled/correlation_unavailable. The root marker is set while live.
- LiveKit: document how Voice finds a call without an SDK CONVERSATION root
  (ingest maps the untyped job_entrypoint root carrying lk.job_id); pin that
  the adapter marks neither job_entrypoint nor agent_session.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…cal policy and create_recording on turns

- create_audio_capture checks the turn_recording policy before it builds the
  AudioBufferProcessor and sets enable_turn_audio only when at least one side
  is admitted; otherwise Pipecat keeps no turn audio, no turn handler is
  registered and turn_status carries the reason.
- Tests: a local CapturePolicy(record_inputs=False) reaches the per-turn
  preflight (output turns only, policy_denied for input); a created recorder's
  turns preflight as create_recording, so revoking server record_audio after
  bind denies later turns (recording_disabled); turn uploads never project
  fi.attachment.refs and bind on association.parent.
- README: per-turn refs and the create-time probe.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
….8.10/1.8.11 worker)

A real livekit-agents worker (loopback livekit-server, explicit dispatch, the
recording binding admitted) shows:
- tracing set up inside the entrypoint (the Quick Start): job_entrypoint is
  not recorded, agent_session is the parentless root of the call's trace and
  the binding marks it CONVERSATION; job_shutdown and room_connect are
  separate root traces.
- tracing set up before the job (setup_fnc): job_entrypoint is the root with
  lk.job_id; agent_session and job_shutdown are its children; not marked.

The README said the inside-entrypoint trace was rooted at job_shutdown and
the call was never listed in Voice; both statements are corrected.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Capture context at native span creation, retain privacy-qualified sources until export, and preserve explicit identifiers. Keep attachment capture independent and avoid process-wide session state. Cover overlapping calls, late children, no-context behavior, both transports and privacy denial.

Part of TH-2379 (Linear).

Co-Authored-By: Claude Code <noreply@anthropic.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant