fix(agents): flush STT on manual commitUserTurn with detached input audio - #2487
fix(agents): flush STT on manual commitUserTurn with detached input audio#2487programmarchy wants to merge 1 commit into
Conversation
`AgentSession.commitUserTurn()` called the activity without options, so `audioDetached` always defaulted to false and the recognizer never pushed the silence that flushes the STT. With the documented push-to-talk release (`input.setAudioEnabled(false)` then `commitUserTurn()`), the provider gets no trailing audio, the final transcript never arrives inside the commit wait, and the turn is committed from the clipped interim text. Pass `audioDetached: !this.input.audioEnabled`, matching Python's `AgentActivity.commit_user_turn`. The silence flush also requires `AudioRecognition.sampleRate`, which was only recorded from VAD start-of-speech events. Record it from every frame forwarded to the STT, as Python's `_push_audio` does, so the flush works with manual turn detection and no VAD. Adds a regression test with a fake streaming STT that finalizes only after trailing audio; it commits "can you check" before this change and "can you check the connection" after. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PHUXLhxd6NUoXwLzHKvjAv
🦋 Changeset detectedLatest commit: 35fb269 The changes in this PR will be included in the next version bump. This PR includes changesets to release 39 packages
Not sure what this means? Click here to learn what changesets are. Click here if you're a maintainer who wants to add another changeset to this PR |
|
|
![devin-ai-integration[bot]](https://avatars.githubusercontent.com/in/811515?s=60&v=4){kind=small}
| } | ||
|
|
||
| this.activity.commitUserTurn(); | ||
| this.activity.commitUserTurn({ audioDetached: !this.input.audioEnabled }); |
There was a problem hiding this comment.
🟡 Recent finals skip detached flush
When a prior segment finalized within 500 ms, commitUserTurn() skips the detached-audio flush. AudioRecognition.commitUserTurn gates both silence and waiting on that prior final's timestamp. Trailing words from the current segment can be clipped.
Learn more
A manual user turn can contain several STT segments. lastFinalTranscriptTime only records when any segment finalized; it does not prove that no audio arrived afterward. When commit follows a recent intermediate final, the timestamp gate bypasses both the new silence frame and the final-transcript wait. The latest segment therefore remains pending while EOU commits the transcript already available.
Example: The provider finalizes “Can you” after a pause. The user immediately says “check the connection” and releases push-to-talk 300 ms later. Commit sees the recent first final, sends no silence, and can commit only “Can you” instead of the complete turn.
Recommended fix: For detached audio, flush and await finalization whenever audio has arrived since the latest final. Track an input-frame generation or timestamp alongside final events, or always perform the detached flush and bounded wait.
Was this helpful? React with 👍 or 👎 to provide feedback.
2 participants
Description
With the documented push-to-talk flow (
session.input.setAudioEnabled(false)followed bysession.commitUserTurn()), the committed user turn is clipped. In a real room with LiveKit Inference STT, speaking "Can you check the connection?" and releasing right after the last word committed "Can you check". The full final transcript only showed up later, during shutdown.Two parity gaps with the Python framework cause this:
AgentSession.commitUserTurn()callsactivity.commitUserTurn()with no options, soaudioDetacheddefaults tofalseandAudioRecognition.commitUserTurnnever pushes the silence frame that flushes the STT. The provider receives no trailing audio, the final never arrives inside the 500 ms commit wait, and the interim text is committed. Python passesaudio_detached=not self._session.input.audio_enabled(agent_activity.py).closeImplalready passesaudioDetached: true, which is why the complete transcript appeared at shutdown.this.sampleRateisundefined, and it was only recorded from VADSTART_OF_SPEECHevents. Python records it from every pushed frame in_push_audio(audio_recognition.py). Without that, fix 1 alone does nothing under manual turn detection withvad: null, or whenever the VAD has not fired for the turn.Changes Made
AgentSession.commitUserTurn()passes{ audioDetached: !this.input.audioEnabled }to the activity, and documents the behavior.AudioRecognition.forwardInputAudioToSttrecordsthis.sampleRatefrom each forwarded frame.agent_session_manual_commit_flush.test.ts: a fake streaming STT that emits an interim mid-speech and the final only after 200 ms of trailing audio. Under manual turn detection withvad: null, push 300 ms of speech, disable input, commit. Onmainthe committed turn is"can you check"; with only the session change it is still"can you check"; with both changes it is"can you check the connection".@livekit/agentspatch).Pre-Review Checklist
pnpm test agents/src/voice/agent_session agents/src/voice/audio_recognition agents/src/voice/agent_activitypasses (33 files, 277 tests); prettier and eslint clean on touched filesTesting
restaurant_agent.ts/realtime_agent.ts: not a major change; the default (non-manual) path is unaffected sinceaudioDetachedstaysfalsewhile input audio is enabledAdditional Notes
Found while building a push-to-talk agent on
@livekit/agents@1.8.1. We are carrying the one-line session change as a pnpm patch; a six-turn real-room lifecycle check (press, release, cancel, empty turn, two segments with a pause, repeat) passes with it, committing about 600 ms after release. The sample-rate change is included here because the defaultinference.VADhappened to report it in our app, but any configuration without a VAD speech event for the turn would still clip.This does not change the 500 ms commit wait or add a
transcript_timeout/stt_flush_durationoption as in Python; those could be a follow-up.🤖 Generated with Claude Code
https://claude.ai/code/session_01PHUXLhxd6NUoXwLzHKvjAv