Initial implementation for transcription deduplication by 1egoman · Pull Request #2093 · livekit/client-sdk-js

1egoman · 2026-09-10T16:22:05Z

This pull request contains the web implementation for the transcription deduplication project.

A high level summary of the project:

Today all transcriptions are sent twice over the reliable data channel, once in legacy format, and once in modern data streams format. Because they are sent twice, the reliable data channel is clogged with data and when an end user is on a poor bandwidth network connection (ie, mobile) this can result in a very poor user experience - RPCs get missed, data stream are only partially delivered, etc.

The way this has implemented is there is a new client protocol version that has been introduced, 3. Any client which advertises this client protocol must:

NOT require the agents sdk to send it legacy transcriptions. Only modern, data stream transcriptions will need to be sent over the data channel. If any legacy data stream transcriptions are sent, they will be ignored.
Back-convert any modern data stream transcriptions into legacy RoomEvent.TranscriptionReceived events. This is the majority of the change - doing this back conversion ended up being more involved than initially expected.

In parallel to this, a change has been made to agents and (soon) to agents-js which stops sending legacy transcriptions to clients with client protocol of 3 or higher.

TODO

Fix this out of date comment: Initial implementation for transcription deduplication #2093 (comment)

changeset-bot · 2026-09-10T16:22:10Z

🦋 Changeset detected

Latest commit: f380c12

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 1 package

Name	Type
livekit-client	Minor

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

github-actions · 2026-09-10T16:24:31Z

size-limit report 📦

Path	Size
dist/livekit-client.esm.mjs	112.13 KB (+0.74% 🔺)
dist/livekit-client.umd.js	121.36 KB (+0.82% 🔺)

1egoman · 2026-09-14T20:16:32Z

I've tested this with livekit/agents#7240 and have confirmed it works end to end - no packets which match packet.value.case === "transcription" are received when the agents framework is using the code in the pull request, and when running on main matching legacy transcription packets are received.

This reverts commit 64e6dab.

This reverts commit 31ebd40.

This reverts commit 09cdf6d.

The agents SDK sets attributes on participants. Each attribute has a string key. This commit adds an enum that contains these keys. The enum is a copy of the enum in components-core. components-core can import this enum and remove its own copy. The generated file attribute-typings.ts also contains these keys. That file comes from the attribute-definitions repository. Code generation cannot use this enum. Thus the keys stay in two places. A comment on the enum tells you which place is correct. No code uses the enum yet.

… types Agents send transcriptions on a reserved data stream topic. This commit adds a constant for the name of that topic. This commit also adds an events file for IncomingDataStreamManager. The file declares one event. The event tells a listener that a transcription stream started. The event carries a reader and the identity of the participant that sent the stream. Managers in this codebase declare their events in a sibling events file. This file obeys that rule. No code uses the constant or the types yet.

…emitter IncomingDataStreamManager must tell other parts of the SDK when a transcription stream starts. An event is the correct tool for this. This commit makes the class extend a typed event emitter. The class uses the callbacks type from the sibling events file. Other managers in this codebase use the same pattern. The constructor now calls super() first. The class does not send an event yet. Behaviour does not change.

A later commit lets a text stream have more than one consumer. Each consumer reads through its own ReadableStream. This commit prepares the data shape for that change. Before this commit, the manager kept one record for each stream. The record held the stream controller and also the stream info, the start time, and the identity of the sender. After this commit, the manager keeps a group for each stream. The group holds the info, the start time, and the sender identity one time. The group also holds a list of controllers. The list has one controller now. Each text stream still has exactly one consumer. Behaviour does not change. REVIEWER ATTENTION: This code is shared. All text streams use it, which includes RPC streams and application streams. Make sure that the behaviour is the same as before.

The SDK must read transcription streams itself. An application can also read the same topic. components-core does this today to get the lk.expression attribute. The SDK must not take the topic away from the application. This commit builds a list of consumers for each text stream. The list holds the application handler for the topic. For the transcription topic, the list also holds a callback that sends the transcriptionStreamArrived event. The manager gives each consumer its own reader. The manager adds the transcription consumer only if a listener exists. If the list is empty, the manager drops the stream and writes a debug log. This is the behaviour from before. The check for a duplicate stream id moves out of the ReadableStream start callback. The check now runs before the manager builds any consumer. The check still throws immediately. REVIEWER ATTENTION: A ReadableStream permits only one consumer. Two consumers of one reader get different parts of the data. Make sure that each consumer gets its own reader.

These tests show that two consumers of one transcription stream both get all of the data. The first test registers an application handler and listens for the event. Both consumers get the full text. The second test sends the payload inside the header packet. This is a single-packet stream. Both consumers get the full text. The third test sends a trailer that sets the lk.transcription_final attribute. Both readers see the new attribute value. This shows that the consumers share one info object.

These tests show when the manager sends the transcriptionStreamArrived event and when it does not. The first test registers no application handler. The event still occurs. This is the usual case, because most applications do not read the transcription topic. The second test uses a different topic. The event does not occur. The third test registers no consumer at all. The manager drops the stream. The test sends the same header two times. Neither call throws. This shows that the manager kept no state for the stream.

Agents send transcriptions on the lk.transcription topic as text streams. This module turns those streams back into the Transcription messages that the legacy data packet carried. The events that applications listen to then stay the same. Two stream shapes arrive on the topic: - Agent speech uses one stream for each segment. Each chunk adds more text. The closing trailer carries the finality attribute. - User speech-to-text uses a new stream for each update. All of the streams share one segment id. Each stream carries the full text. The header carries the finality attribute. Thus a new stream id for a segment that is already open replaces the text. It does not add to it. The module trusts the lk.transcription_final attribute when the attribute is present. The end of a stream means final only when the attribute is absent. If the module made every stream end final, it would wrongly finalize each interim user transcript. The module has no dependency on Room or on RTCEngine. It reports each update through a callback. REVIEWER ATTENTION: The rule for replace, and the three states of the finality attribute, are the most difficult parts of this change. Read the loop in handleTextStream and the isFinal method together.

This commit adds the test harness for the converter. The harness builds a reader that a test drives step by step. A test can write a chunk, look at the result, and then close the stream. The close merges attributes into the info object. The real manager does the same thing when a trailer arrives. Three tests use the harness: - Chunks of one stream add together. The closing attribute makes the segment final. - A second stream with the same segment id replaces the text. It does not add to the text. - A close does not send the same update two times.

These tests show how the converter decides that a segment is final, and how it keeps segments apart. - A stream that ends without the finality attribute gives a final segment. - The converter accepts the string "true" and the string "1". Agents send the string form. - If the header has no segment id, the converter uses the stream id instead. - Two senders that use the same segment id do not mix their text together.

These tests show what the converter does at the end of the life of a segment. - A stream that carries no text gives no update at all. - A stream that fails in the middle still closes its segment. The segment becomes final with the text that arrived before the failure. - reset() removes the state of segments that are still open. The test uses the same stream id two times. Without reset() the text would add together and the test would fail.

The legacy data packet always told the application which participant spoke and which track carried the speech. A transcription stream does not always give the same two facts. This commit fills the gaps. The converter now takes two more callbacks. A later commit gives it the versions that read from Room. For the track sid, the converter tries three sources in this order: 1. the lk.transcribed_track_id attribute 2. the microphone track of the participant that it selected 3. the microphone track of the participant that sent the stream For the speaker, the converter looks for a participant that publishes for the sender. An avatar worker does this. The legacy packet named that worker, not the agent, so the converter does the same. The worker is also the participant that publishes the audio track, so a publication can be found. The test harness now supplies the two new callbacks. The tests for them come next. REVIEWER ATTENTION: The chain uses || and not ??. A callback that reports "nothing found" as an empty string must fall through to the next source. An empty identity finds no participant, and an empty sid finds no publication. Either one stops the track events without a warning.

These tests show the order of the three sources for the track sid, and the choice of the speaker. - The attribute wins when it is present. - Without the attribute, the microphone track of the speaker is used. - For an avatar session, the converter names the worker that publishes for the agent. It also uses the microphone track of that worker. - If the worker has no microphone track, the converter uses the track of the sender. - If nothing is found, the track sid stays empty. - A callback that returns an empty string means "nothing found". The converter goes on to the next source. It does not stop.

An agent can run with the json_format option. The agent then wraps each write as a JSON TimedString object. The text sits in a text field of that object. Nothing on the wire tells the receiver which form a stream uses. There is no attribute and no special mime type. Thus the converter must examine the payload. The converter parses a chunk only if the chunk starts with a brace and ends with a brace. If the parse gives an object with a text field of type string, the converter uses that text. In all other cases the converter uses the chunk without a change. This includes a chunk that fails to parse. The converter does not read the times from the TimedString object. The proto fields are unsigned integers, but the TimedString fields are decimal numbers in an unknown unit. The legacy packet always sent zero. A wrong unit is worse than no value.

These tests show that the converter reads both payload forms correctly. The first test sends JSON TimedString chunks. The converter takes only the text field. The times in the segment stay zero. The second test sends two chunks that look like JSON but are not. One chunk has no closing brace. The other chunk parses but has no text field. The converter keeps both chunks without a change, and it keeps the space at the start of the second chunk. If the converter used the trimmed text, real transcripts would lose spaces.

…ption The method took a participant, but it never used it. The name had an underscore in front to show this. The method finds the participant from the transcribedParticipantIdentity field of the message instead. The parameter is also not a value to bring back later. For a stream, the sender is the participant that spoke. For a legacy packet, the sender is the agent. The two channels do not agree on the meaning, so the value would confuse a reader. Behaviour does not change.

…treams Room now builds a TranscriptionStreamConverter and listens to the transcriptionStreamArrived event of the data stream manager. Each transcription stream goes to the converter. The converter reports each update, and Room sends it to the existing handleTranscription method. The three transcription events keep their current shape. Room supplies the two callbacks that the converter needs: - getMicrophoneTrackSid finds the microphone track of a participant. - getDelegatingPublisherIdentity finds a participant that publishes for another participant. An avatar worker does this. Both callbacks return undefined when they find nothing. They must never return an empty string. Room also clears the converter state when the room disconnects. Room still handles legacy transcription packets. Thus an agent that sends both channels causes two events for one utterance. The next commit removes the legacy path. REVIEWER ATTENTION: Room listens to the event. Room does not register a handler for the lk.transcription topic. This is deliberate. An application that reads that topic keeps its access. components-core needs this to read the lk.expression attribute.

Room no longer sends events for a legacy Transcription data packet. It writes a debug log and stops. All transcription events now come from the lk.transcription streams. This removes the second copy of each utterance that the previous commit created. REVIEWER ATTENTION: This change takes effect as soon as a user upgrades. It does not wait for agents to use client protocol 3. Agents today send both channels, so this release starts to read the stream channel and starts to drop the legacy packets. A service that is not an agent and that sends legacy Transcription packets also stops working. This SDK has no method to send those packets, so no client-to-client case exists.

This commit adds a test fixture that builds a connected Room with one remote participant. That participant publishes a microphone track. Helper functions push data packets into the room and send a complete transcription stream. Two tests use the fixture: - A legacy Transcription data packet gives no events at all. - A transcription stream gives a RoomEvent.TranscriptionReceived event and a matching event on the track publication. The event names the correct participant.

Two more tests use the fixture from the previous commit: - A stream without the lk.transcribed_track_id attribute still gives a track event. Room finds the microphone track of the speaker instead. - An application that registers a handler for the lk.transcription topic still receives the full text. This test also guards the design: registerTextStreamHandler throws if a handler for a topic already exists. If Room registered that topic for itself, the call in this test would throw and the test would fail.

…eam support The client protocol number tells other participants what this client can do. This commit adds the value 3 and makes the client advertise it. Value 3 means two things. As a client, it means "I build transcription events from the lk.transcription streams, so do not send me the legacy Transcription packet". As an agent, it means "I send every transcription on the lk.transcription stream". An agents SDK change will use this value to stop the second copy. That change is not in this repository. The commit adds a test for the value. The test also shows that the advertised value points at the new constant. REVIEWER ATTENTION: This is a wire contract. The three older constants keep their values. Every comparison in the repository uses "greater than or equal to", so the new value breaks nothing. This commit comes last on purpose: agents must not stop sending legacy packets until all of the code above works.

The changeset asks for a minor release of livekit-client. The text tells users that the change affects them as soon as they upgrade. It does not wait for agents to use client protocol 3. The text also tells users that an application which reads the lk.transcription topic directly keeps working.

devin-ai-integration

Devin Review found 2 potential issues.

2 flags not posted on this PR by your GitHub settings — view them in Devin Review. (Configure)

1egoman · 2026-09-14T20:31:37Z

+        // The transcription tap runs alongside the application handler, each with its own reader,
+        // so the SDK can rebuild transcription events without taking the reserved topic away from
+        // an application that reads it too. `listenerCount` keeps the "nobody wants this stream,
+        // drop it" behavior intact when no one has subscribed.
+        const streamHandlerCallbacks: Array<TextStreamHandler> = [];
+        if (
+          streamHeader.topic === TRANSCRIPTION_TOPIC &&
+          this.listenerCount('transcriptionStreamArrived') > 0
+        ) {
+          streamHandlerCallbacks.push((reader, { identity }) => {
+            this.emit('transcriptionStreamArrived', { reader, participantIdentity: identity });
+          });
+        }


Note to reviewers - this "side channel" transcriptionStreamArrived event is how other subsystems tap off of IncomingDataStreamManager without requiring a whole separate subscription management system for internal subscribers.

1egoman · 2026-09-14T20:34:56Z

+export default class TranscriptionStreamConverter {
+  private log = log;
+
+  private options: TranscriptionStreamConverterOptions;
+
+  /** In-flight segments, keyed by `partialKey(senderIdentity, segmentId)`. */
+  private partials = new Map<string, PartialTranscription>();
+
+  constructor(options: TranscriptionStreamConverterOptions) {
+    this.options = options;
+  }
+
+  /** Drops all in-flight segment state. */
+  reset() {
+    this.partials.clear();
+  }
+
+  handleTextStream = async (reader: TextStreamReader, senderIdentity: string) => {


Note to reviewers - this net new TranscriptionStreamConverter class which does the modern -> legacy transcription conversion is really the high risk part of this change, and is where the most scrutiny would be appreciated, especially from those who are familiar with the peculiars of how the legacy transcription system works.

1egoman · 2026-09-14T20:40:50Z

+    this.transcriptionStreamConverter = new TranscriptionStreamConverter({
+      onTranscription: (transcription) => this.handleTranscription(transcription),
+      getMicrophoneTrackSid: this.getMicrophoneTrackSid,
+      getDelegatingPublisherIdentity: this.getDelegatingPublisherIdentity,
+    });
+    this.incomingDataStreamManager.on(
+      'transcriptionStreamArrived',
+      ({ reader, participantIdentity }) => {
+        this.transcriptionStreamConverter.handleTextStream(reader, participantIdentity);
+      },
+    );


Note to reviewers - here is where everything is hooked together:

The new IncomingDataStreamManager event (transcriptionStreamArrived) is wired up to TranscriptionStreamConverter via this.transcriptionStreamConverter.handleTextStream(...)

this.handleTranscription(...) is called with the newly generated legacy transcriptions.

1egoman · 2026-09-14T20:41:31Z

+      // Legacy `Transcription` packets are ignored: transcription events are rebuilt from the
+      // `lk.transcription` data stream channel instead, which this client advertises support for
+      // via client protocol 3. See
+      // docs/superpowers/specs/2026-09-04-transcription-back-conversion-design.md
+      this.log.debug('ignoring legacy transcription data packet', this.logContext);


Note to self - drop docs/superpowers/specs/2026-09-04-transcription-back-conversion-design.md from this comment before merging

… cleared clearControllers() removed each stream from its map, but it did not close or fail the stream. A consumer that waited for the next chunk thus waited for ever. The promise never settled, so the reader stayed in memory together with everything that the consumer held. The manager now fails every open text and byte controller before it clears the maps. Each consumer gets a DataStreamError with the AbnormalEnd reason. This fault is older than the transcription work, but that work makes the fault occur much more often. The SDK now opens a reader for each transcription stream. Before, only an application handler opened one. A room that disconnects in the middle of an utterance therefore held the Room object through the callbacks of the converter. Room calls reset() on the converter before it calls clearControllers(). The converter thus finds no open segment when the failure arrives, and it sends no late final event. The order is already correct and does not change. REVIEWER ATTENTION: An application that reads a data stream sees a change. A read that waited for ever now fails with an error. This is the better result, and the code already fails readers in the same way when the participant that sends the stream disconnects.

…cleared Two tests show that clearControllers() settles the streams that are open. The first test opens one text stream and one byte stream, and starts a read on each. It then clears the controllers. Both reads fail with an error. The second test opens a transcription stream and starts a read through the tap event. It then clears the controllers. The read fails with an error. This is the case that a room disconnect in the middle of an utterance creates. Both tests fail if clearControllers() only empties the maps. The reads then wait for ever and the tests stop at the time limit.

1egoman added 3 commits September 4, 2026 13:06

feat: commit temporary transcription deduplication spec

09cdf6d

feat: update transcription deduplication spec

31ebd40

feat; add transcription deduplication plan

64e6dab

1egoman commented Sep 10, 2026

View reviewed changes

Comment thread src/room/data-stream/incoming/IncomingDataStreamManager.ts

1egoman force-pushed the deduplicate-transcriptions branch from 9a0ad47 to f3b390f Compare September 10, 2026 19:07

1egoman mentioned this pull request Sep 11, 2026

Only send modern transcriptions to client sdks which can auto back convert modern -> legacy transcriptions internally livekit/agents#7240

Open

1egoman marked this pull request as ready for review September 14, 2026 20:16

1egoman requested review from lukasIO and xianshijing-lk as code owners September 14, 2026 20:16

1egoman added 17 commits September 14, 2026 16:18

Revert "feat; add transcription deduplication plan"

ca3086d

This reverts commit 64e6dab.

Revert "feat: update transcription deduplication spec"

1370d3b

This reverts commit 31ebd40.

Revert "feat: commit temporary transcription deduplication spec"

3aa7f21

This reverts commit 09cdf6d.

1egoman added 8 commits September 14, 2026 16:18

1egoman force-pushed the deduplicate-transcriptions branch from eb6ae7d to 9443444 Compare September 14, 2026 20:18

devin-ai-integration Bot reviewed Sep 14, 2026

View reviewed changes

Comment thread src/room/Room.ts

Comment thread src/room/transcription/TranscriptionStreamConverter.ts

refactor: add docstrings for TextStreamControllerGroup

f8e5a1f

1egoman commented Sep 14, 2026

View reviewed changes

1egoman added 2 commits September 15, 2026 11:13

Navigation Menu

Uh oh!

Initial implementation for transcription deduplication - #2093

Initial implementation for transcription deduplication#2093
1egoman wants to merge 31 commits into
mainfrom
deduplicate-transcriptions

1egoman commented Sep 10, 2026 •

edited

Loading

Uh oh!

changeset-bot Bot commented Sep 10, 2026 •

edited

Loading

Uh oh!

github-actions Bot commented Sep 10, 2026 •

edited

Loading

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

1egoman commented Sep 14, 2026

Uh oh!

devin-ai-integration Bot left a comment

Uh oh!

Uh oh!

Uh oh!

1egoman Sep 14, 2026

Uh oh!

1egoman Sep 14, 2026

Uh oh!

1egoman Sep 14, 2026

Uh oh!

1egoman Sep 14, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Uh oh!

Conversation

1egoman commented Sep 10, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

TODO

Uh oh!

changeset-bot Bot commented Sep 10, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🦋 Changeset detected

Uh oh!

github-actions Bot commented Sep 10, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

size-limit report 📦

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

1egoman commented Sep 14, 2026

Uh oh!

devin-ai-integration Bot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

1egoman Sep 14, 2026

Choose a reason for hiding this comment

Uh oh!

1egoman Sep 14, 2026

Choose a reason for hiding this comment

Uh oh!

1egoman Sep 14, 2026

Choose a reason for hiding this comment

Uh oh!

1egoman Sep 14, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

1egoman commented Sep 10, 2026 •

edited

Loading

changeset-bot Bot commented Sep 10, 2026 •

edited

Loading

github-actions Bot commented Sep 10, 2026 •

edited

Loading