Repository navigation
feat(python): Pipecat recorder, LiveKit owned-file and Egress import on the shared attachment core - #678
Draft
nik13 wants to merge 10 commits into
Draft
feat(python): Pipecat recorder, LiveKit owned-file and Egress import on the shared attachment core#678nik13 wants to merge 10 commits into
nik13 wants to merge 10 commits into
Conversation
Keep Pipecat streams and completed Egress imports fail-closed until the shared transport prerequisites are qualified. Co-Authored-By: Claude Code <noreply@anthropic.com>
Co-Authored-By: Claude Code <noreply@anthropic.com>
… import on the shared core V01: bind_audio_processor observes the app's AudioBufferProcessor (on_audio_data / on_track_audio_data) into a core RecordingWriter bound to the conversation span Pipecat's own TurnTraceObserver started; mixed, stereo and separate_tracks manifests; recording stays off unless enabled; create_audio_capture needs local and server record_audio. V02: LiveKit session binding uses public preflight, identify() and attach_owned_recording (tier single, direction mixed); no private manager or scope internals. V03: LiveKitEgressResolver + async import_completed_egress through the core import_completed_recording seam (completed audio-only Egress only). fi.span.kind=CONVERSATION is set only on a live, parentless SDK span. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…t-in voice recordings Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The runner apt mirror failed DNS on both attempts. The synthetic Ogg/Opus fixture is now encoded with PyAV's bundled codecs (already a LiveKit Agents dependency), so the apt step is removed. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ce root docs - Pipecat per-turn uploads (opt-in): `turn_audio=TurnAudioCaps(max_turns=, max_bytes=, max_duration_ms=)` on bind_audio_processor/create_audio_capture. on_user_turn_audio -> input, on_bot_turn_audio -> output, purpose turn_recording, bound to Pipecat's own turn span via TurnTraceObserver.get_turn_context(turn_number). Caps are per call, checked on length before any copy; an over-cap turn is skipped whole (queue_full for the count, too_large for bytes/duration). enable_turn_audio without caps is still refused as unbounded_source. No fallback to the conversation or an ambient span. Turn handlers are not registered when both sides are denied. - Short calls: the conversation identity is snapshotted while Pipecat's conversation span is live (bind, on_recording_started, every chunk), so a call shorter than buffer_size whose only chunk runs after the worker ended the span still binds to that conversation instead of coming out unsampled/correlation_unavailable. The root marker is set while live. - LiveKit: document how Voice finds a call without an SDK CONVERSATION root (ingest maps the untyped job_entrypoint root carrying lk.job_id); pin that the adapter marks neither job_entrypoint nor agent_session. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…cal policy and create_recording on turns - create_audio_capture checks the turn_recording policy before it builds the AudioBufferProcessor and sets enable_turn_audio only when at least one side is admitted; otherwise Pipecat keeps no turn audio, no turn handler is registered and turn_status carries the reason. - Tests: a local CapturePolicy(record_inputs=False) reaches the per-turn preflight (output turns only, policy_denied for input); a created recorder's turns preflight as create_recording, so revoking server record_audio after bind denies later turns (recording_disabled); turn uploads never project fi.attachment.refs and bind on association.parent. - README: per-turn refs and the create-time probe. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
….8.10/1.8.11 worker) A real livekit-agents worker (loopback livekit-server, explicit dispatch, the recording binding admitted) shows: - tracing set up inside the entrypoint (the Quick Start): job_entrypoint is not recorded, agent_session is the parentless root of the call's trace and the binding marks it CONVERSATION; job_shutdown and room_connect are separate root traces. - tracing set up before the job (setup_fnc): job_entrypoint is the root with lk.job_id; agent_session and job_shutdown are its children; not marked. The README said the inside-entrypoint trace was rooted at job_shutdown and the call was never listed in Voice; both statements are corrected. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Capture context at native span creation, retain privacy-qualified sources until export, and preserve explicit identifiers. Keep attachment capture independent and avoid process-wide session state. Cover overlapping calls, late children, no-context behavior, both transports and privacy denial. Part of TH-2379 (Linear). Co-Authored-By: Claude Code <noreply@anthropic.com>
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Part of TH-2379 (Linear).
Summary
Draft, head
1d02a461(review follow-ups on77f18b26, see "Review follow-ups"; earlier follow-ups on74eb15b1: Pipecat per-turn audio, short-call binding, LiveKit Voice root docs). Stacked on #676 (feat/attachments-python-core). This merges core head5a7e1105and moves every voice adapter onto the shared core API: preflight,start_recording, segments plus manifests,attach_owned_recordingandimport_completed_recording. Attachments stay off unless enabled. Creating a new recording is separately off.V01: Pipecat recorder (
traceai_pipecat.attachments)bind_audio_processor(processor, AudioSessionContext.from_pipeline(worker), manager=..., layout="auto"|"separate_tracks")observes the app's existingAudioBufferProcessor.on_audio_data, or toon_track_audio_dataforseparate_tracks, and toon_recording_stopped.RecordingWriter, which writes WAV segments of up to 30 s and publishes a manifest with gaps.Bound to the real conversation. The recording attaches to the
conversationspan that Pipecat's ownTurnTraceObserverstarted; turn spans are its children. The writer opens on the first chunk, so a conversation span that starts asynchronously is still found.conversation.idasconversation_id.correlation_unavailableand nothing is kept.Layouts:
mixedstereoseparate_tracksseparate_tracksThe core evaluates
mixedandstereoasmixed, so any hide flag denies the whole recording. Withseparate_tracks, a hidden side becomes apolicy_deniedgap for that track and the other side still uploads.Each start/stop cycle is its own recording.
on_recording_stoppedfinishes the cycle in the background.await binding.finish()awaits uploads within the deadline (at most 30 s).Bounded.
0 < buffer_size <= 128 KiBis required.buffer_size=0returnsunbounded_source; so doesenable_turn_audiounless per-turn uploads are opted in with explicit caps (below).queue_fullgap rather than memory growth (window of 2 segments).create_audio_capture(...)returns a configured, unstarted processor only when both local and serverrecord_audioallow it (operation="create_recording").streaming_unavailableis retired. Support set is exactlypipecat-ai==1.15.0. 1.0.0 returnsunsupported_version(checked on the VM).Follow-ups: per-turn audio, short calls, LiveKit Voice root
bind_audio_processor(..., turn_audio=TurnAudioCaps(max_turns=, max_bytes=, max_duration_ms=))(alsocreate_audio_captureandtraceai.bind_pipecat_audio_processor). All three caps are required, per call, with ceilings 1000 turns / 64 MiB / 2 h.on_user_turn_audiouploads asinput,on_bot_turn_audioasoutput: mono WAV, purposeturn_recording,turn_id= Pipecat turn number, the call'sconversation_id.TurnTraceObserver.get_turn_context(turn_number)(public API). Never the conversation span or an ambient span; an explicit-span context has no turn spans, so turns endcorrelation_unavailable.fi.attachment.refs(or anyfi.attachment.*attribute) onto the turn span, even while it is open: the preflight target is aCorrelationHandle. Ingest binds it onassociation.parent, which names the turn span.preflight(deny-wins, per side) -> only then the WAV copy ->attach. An over-cap turn is skipped whole:queue_full(count) ortoo_large(bytes/duration). Never truncated.turn_recordingon input/output: if both sides are denied, no turn handler is registered (turn_status = policy_denied).enable_turn_audio=Truewithout caps staysunbounded_source; caps with a processor lackingenable_turn_audioraiseValueError.binding.turn_status,binding.turn_denials(counts by reason),binding.turn_handles;await binding.finish()also waits for turn uploads within the same deadline.buffer_size) no longer come outunsampled/correlation_unavailable. Pipecat runson_audio_dataas a task, so a short call's only chunk arrives on stop and can run after the worker's cleanup ended and cleared the conversation span. The adapter now snapshots the conversation'sCorrelationHandlewhile the span is live (at bind, onon_recording_started, on every chunk) and binds to it when the span is gone. No deferred target was added to the core (a snapshot taken from a live, sampled span is the documentedCorrelationHandleuse). On the snapshot path the conversation span has no carrier stamp; ingest binds on the authenticatedassociation.parentplus the stored span, so refs are not needed. The root marker is set at snapshot time (live, sampled, parentless span only).1d02a461from a real worker run, see below). The SDK marksagent_sessiononly when it is already the trace root. With tracing set up inside the entrypoint (the README Quick Start), it is: the binding marks itCONVERSATION. With tracing set up before the job,job_entrypoint(withlk.job_id) is the root andagent_sessionits child, so nothing is marked and ingest maps the untyped, parentlessjob_entrypointwhen attachments are enabled on the server. A test pins the child shape (neither span getsfi.span.kind/gen_ai.span.kind).V02: LiveKit session recording (
traceai_livekit.attachments)manager._session,manager.policy.evaluate, thestatusproperty,scope._fd).preflight(1-byte ticket, released at once).RecordingFileScope.identify(), plus a deny-onlylstatcheck that refuses hard links and foreign owners.attach_owned_recording(..., direction="mixed"), tiersingle, which matches TS feat(voice): TS LiveKit Agents recording, Pipecat recording link, ElevenLabs speech attachments #685.V03: completed LiveKit Egress (
traceai_livekit.recording)LiveKitEgressResolver(design 5.4).import_completed_egress(...)is nowasyncand goes through the core'simport_completed_recording.EGRESS_COMPLETE, an audio-only request (room composite withaudio_only, or a track composite with no video), and exactly one file result with known size and duration.declared_duration_ms = ceil(ns / 1e6), directionmixed, andregistered_output_idfrom the injectedEgressOutputRegistry(CQ-4).EgressInfocomes from: an object the app already holds, or ONE fixed-IDlist_egress(ListEgressRequest(egress_id=...))through the app's client.Conversation root marker
fi.span.kind=CONVERSATION(plusgen_ai.span.kind, so the Pipecat mapper'sCHAINdefault doesn't contradict it) is set only on a live, sampled SDK span with no parent (local or remote), and only when the binding is admitted. For Pipecat it is set when the conversation identity is snapshotted (at bind if the span already exists, on recording start, or on a chunk), while the span is still live.conversationspan is marked when it is a true root.agent_sessionis the parentless root and the binding marks it. Before the job,agent_sessionis a child ofjob_entrypointand is not marked.Auto module (
traceai)bind_pipecat_audio_processor(processor, context, **options)passes options through.async import_livekit_completed_egress(...)uses the provider's shared manager.bind_pipecat_audio_processor(..., turn_audio=traceai_pipecat.TurnAudioCaps(...))passes the per-turn opt-in through.attachment_status()["adapters"]now reports the binding's own status, so a denied binding is no longer reported assession_bound.Review follow-ups on
1d02a461(two commits on77f18b26)create_audio_capture(..., turn_audio=...)checks the turn policy first. Theturn_recordingprobe (operationcreate_recording, input and output) runs BEFORE theAudioBufferProcessoris built.enable_turn_audio=Trueis set only when at least one side is admitted; otherwise Pipecat keeps no turn audio, no turn handler is registered andturn_statusis the reason (for examplepolicy_denied).New regression tests (the review's surviving mutations):
CapturePolicy(record_inputs=False)withseparate_tracks+turn_audiouploads only the output turn and recordspolicy_deniedfor the input turn (no env flag involved);turn_audio: turn 1 uploads both sides, then the server'srecord_audiois turned off and turn 2 is denied on both sides (recording_disabled), because turns preflight ascreate_recording;enable_turn_audio,turn_status=policy_denied, no turn handlers;fi.attachment.*attribute on the turn span;association.parentis the turn span.Real LiveKit worker (livekit-agents 1.8.11 and 1.8.10, loopback
livekit-server --dev, explicit agent dispatch,AgentSession(...).start(record=True),bind_session(..., owned_recording_files=True)against a loopback TLS F01 server, spans exported per span end):agent_sessionagent_session(session spans),job_shutdown(+room_disconnect,session_end_upload,shutdown_callback),room_connect,event_loop_blocked;job_entrypointnot recordedlk.job_id,fi.span.kind=CONVERSATIONsession_bound,root_marked=True, Ogg uploaded (validating)setup_fnc)job_entrypoint(lk.job_id);agent_sessionandjob_shutdownare its childrensession_bound,root_marked=False, Ogg uploaded (validating)Identical on 1.8.10 and 1.8.11. The README said the inside-entrypoint trace was rooted at
job_shutdownand the call could not be listed; both are corrected.1d02a461python/tests/attachments+python/traceai/tests77f18b26adapter77f18b26already had, now with tests that discriminate)capture_policy(local-policy test), dropsoperation(record_audio revocation test), recorder built withenable_turn_audiobefore the probe (create-time test)DisabledAttachmentManager, poisoned args,turn_audio)disabled, nothing touched, 0 owned bytesOn the first VM run the revocation test failed intermittently on Python 3.11 (4 of 5 runs; 3.12 passed): revoking
record_audioalso stopped turn 1's still-queued output upload (deny-wins on queued work, correct behaviour). The test now waits for turn 1's uploads before revoking.Verification of the follow-ups on
77f18b26(private GCP VM, deleted; synthetic audio, core F01 transport double)python/tests/attachments+python/traceai/tests74eb15b1adapter (plus aTurnAudioCapsstub so the module imports)on_recording_startedsnapshot; count / bytes / duration cap removed; turn falls back to the conversation span; wrong purpose; sides swapped; turn handlers registered when both sides are denied;enable_turn_audioaccepted without caps;finish()ignores turn uploadsDisabledAttachmentManager, poisoned arguments, with and withoutturn_audio)disabled, nothing touched, 0 owned bytesCI on
77f18b26: 13 of 13 green (8 voice component jobs incl. Pipecat 1.15.0 on 3.11/3.12, plus 5 attachment jobs on Python 3.10 to 3.14).Verification (private GCP VM, synthetic audio, core F01 transport double + loopback TLS server)
Exact pins, one venv each (freezes in evidence):
python/tests/attachments+python/traceai/tests(merge is clean)5154dee7) + the new testsstreaming_unavailable.unsupported_versionReal-pipeline proof (V01). A real
PipelineWorker(enable_tracing=True, enable_turn_tracking=True)runs on Pipecat 1.15.0 with synthetic tone frames andauto_start_recording. For mono, stereo andseparate_tracks:separate_tracks);duration_msandcapture_completeare correct;UploadRequestparent is the conversation span Pipecat started;fi.span.kind=CONVERSATIONand the attachment carrier stamp, and no other span does.With an app span as parent, the conversation span is not marked.
Mutations (one line each, run against the head's tests): 9 of 9 detected:
separate_tracksprobed asmixed; unbounded upstream accepted; no finish on stop.EGRESS_ACTIVEtreated as ready; video Egress accepted.Logs and freezes are under
evidence/c2-py-voice/in the coordinator workspace (not in this repository). The VM was deleted.CI on
74eb15b1: 13 of 13 green. That is 8 voice component jobs plus 5 attachment jobs.878db07f, every voice job timed out inapt-get install ffmpegbecause the runners could not resolve the Ubuntu mirror (DNS), on both attempts.74eb15b1encodes the synthetic Ogg/Opus fixture with PyAV's bundled codecs and drops the apt step.pip checkclean.Not done / remains open (all fail closed)
READERSis empty, so V03 is proven against the core's syntheticsubmit_importonly.EgressOutputRegistryimplementation (how an app learnsregistered_output_id) is backend/contract work. Without a registry, the status issource_unavailable.agent_session); without one, nothing marks a root. With tracing before the job, listing relies on the server'sjob_entrypointmapping, which the collector also needs the attachments flag for. Traces ingested before the server flag are not re-classified. Listing and playback in the product are not proven end to end here (Q02).attach_owned_recordingtiersinglesends norecording_id, so the server's recording projection has no manifest and shows it as pending. Either the SDK emits a one-segment manifest for single files or the server synthesizes one; this needs a cross-lane decision and is not changed here.unsupported_version).fi-instrumentation-otelfloor must be raised in the release PR that first publishes a core with attachments (the same caveat as feat(python): attachment core with bounded lifecycle, preflight, recordings and imports #676).E2E coverage
not applicable (no product
e2e/harness in this repository) · flows hit: n/a · marker: n/a · verdict: not applicable. Product voice playback is the separate Q02 / OBS-E2E-061 gate, not a skipped pass.🤖 Generated with Claude Code