feat(rime): support WebSocket v1 streaming

[naszzz]

Summary

• Add Rime WebSocket v1 streaming through websocketURL, with binary protobuf and canonical protobuf JSON from @rimelabs/api@0.0.1.
• Select the model from the endpoint. Send complete sentences in one synthesis context, with local flush, cancellation, connection reuse, and all six audio formats.
• Keep HTTP synthesis and legacy WS3 with aligned timestamps. Set consistent sample rates, validate endpoints, and remove private provider and transport details from errors.
• Keep each Rime stream's options, connection pool, and metrics model fixed after updateOptions(). Handle terminal Rime HTTP TTS errors through the existing error event without an unhandled background rejection.
• Keep implementation and test changes in the Rime plugin. Generate audio samples during test setup and remove the JSON fixtures. Update the README, Rime API report, and Rime patch changeset.

Python counterpart: livekit/agents#6978

Interface and data flow

flowchart TB
    subgraph MAIN["Before this PR"]
        M_TTS["rime.TTS"]
        M_MODE{"useWebsocket / baseURL"}
        M_HTTP["HTTP synthesis"]
        M_WS3["Legacy WS3<br/>JSON messages + aligned timestamps"]

        M_TTS --> M_MODE
        M_MODE -->|"HTTP mode"| M_HTTP
        M_MODE -->|"WebSocket mode"| M_WS3
    end

    subgraph BRANCH["This PR"]
        TTS["rime.TTS"]
        SELECT{"Endpoint configuration"}
        TTS --> SELECT

        SELECT -->|"No websocketURL"| LEGACY{"Legacy mode"}
        LEGACY -->|"HTTP URL"| HTTP["ChunkedStream<br/>HTTP synthesis"]
        LEGACY -->|"WebSocket baseURL or<br/>useWebsocket=true"| WS3["SynthesizeStream + RimePool<br/>WS3 JSON + aligned timestamps"]

        SELECT -->|"websocketURL"| MODEL{"resolveOptions<br/>model resolution"}
        MODEL -->|"/coda/ws"| CODA["modelId = coda"]
        MODEL -->|"/mist/ws"| MIST["modelId = mistv3"]
        MODEL -->|"/ws"| DEDICATED["Dedicated endpoint<br/>explicit modelId required"]
        CODA --> V1
        MIST --> V1
        DEDICATED --> V1

        V1["SynthesizeStream + RimePool<br/>sentence tokenizer + connection reuse"]
        V1 --> INPUT["pushText: send complete sentences<br/>flush: release buffered text locally<br/>endInput: send end<br/>close: cancel active context"]
        INPUT --> PROTOCOL{"websocketProtocol"}
        PROTOCOL -->|"binary, default"| BINARY["rime.v1.binary<br/>protobuf frames"]
        PROTOCOL -->|"json"| JSON["rime.v1.json<br/>canonical protobuf JSON"]
        BINARY --> WIRE
        JSON --> WIRE
        WIRE["RimeConnection<br/>ready, start, text*, end or cancel<br/>started, audio*, done"]
        WIRE --> AUDIO["RimeAudio<br/>PCM, PCMU, WAV, MP3, Ogg Opus, WebM Opus<br/>mono PCM frames at samplingRate"]
    end

    subgraph MODEL_STATE["Endpoint and model safety"]
        UPDATE["updateOptions"]
        UPDATE --> IDENTITY{"Model changed on the same<br/>normalized model endpoint?"}
        IDENTITY -->|"yes"| REJECT["Reject update"]
        IDENTITY -->|"no"| ALLOW["Accept valid options<br/>transport mode stays fixed"]
        ALLOW --> CONNECTION{"Full URL, API key,<br/>or protocol changed?"}
        CONNECTION -->|"yes"| POOL["Replace RimePool<br/>retire old pool after its streams finish"]
        CONNECTION -->|"no"| KEEP["Keep RimePool"]
        NORMAL["Model endpoint identity ignores query and trailing slash<br/>Normalizes scheme, host, and effective port<br/>Endpoint validation rejects fragments and user information"]
        NORMAL -.-> IDENTITY
    end

    subgraph STREAM_STATE["Per-stream state"]
        CURRENT["Current TTS options and RimePool"]
        CURRENT --> NEW["Create SynthesizeStream"]
        NEW --> SNAPSHOT["Copy options and retain pool<br/>Rime StreamTTS preserves stream metadata"]
        SNAPSHOT --> METRICS["Existing streams keep their model and endpoint<br/>after later TTS updates"]
        SNAPSHOT --> EVENTS["Forward metrics and errors<br/>to the owning Rime TTS"]
    end

    TTS -.-> UPDATE
    TTS -.-> CURRENT

Loading

The JS implementation uses one SynthesizeStream for v1 and WS3, with RimeConnection for transport and RimeAudio for decoding. V1 always uses sentence input. Its flush() keeps the synthesis context open. V1 does not provide aligned timestamps.

Testing

• pnpm exec vitest run plugins/rime/src: 136 passed, 1 skipped. The service test requires API keys; live provider behavior was not verified.
• Builds for @livekit/agents and @livekit/agents-plugin-rime, including TypeScript declarations: passed.
• API checks for both packages: passed.
• Rime lint, formatting checks for changed files, and git diff --check: passed.
• Audio tests generate all six formats. Compressed sample generation requires a full FFmpeg build with libmp3lame and libopus on PATH.

[changeset-bot]

🦋 Changeset detected

Latest commit: 9d9f0e3

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 38 packages

Name	Type
@livekit/agents-plugin-rime	Patch
@livekit/agents	Patch
@livekit/agents-plugin-anam	Patch
@livekit/agents-plugin-anthropic	Patch
@livekit/agents-plugin-assemblyai	Patch
@livekit/agents-plugin-azure	Patch
@livekit/agents-plugin-baseten	Patch
@livekit/agents-plugin-bey	Patch
@livekit/agents-plugin-cartesia	Patch
@livekit/agents-plugin-cerebras	Patch
@livekit/agents-plugin-deepgram	Patch
@livekit/agents-plugin-did	Patch
@livekit/agents-plugin-elevenlabs	Patch
@livekit/agents-plugin-fishaudio	Patch
@livekit/agents-plugin-google	Patch
@livekit/agents-plugin-hume	Patch
@livekit/agents-plugin-inworld	Patch
@livekit/agents-plugin-krisp	Patch
@livekit/agents-plugin-lemonslice	Patch
@livekit/agents-plugin-liveavatar	Patch
@livekit/agents-plugin-livekit	Patch
@livekit/agents-plugin-minimax	Patch
@livekit/agents-plugin-mistral	Patch
@livekit/agents-plugin-mistralai	Patch
@livekit/agents-plugin-neuphonic	Patch
@livekit/agents-plugin-openai	Patch
@livekit/agents-plugin-perplexity	Patch
@livekit/agents-plugin-phonic	Patch
@livekit/agents-plugin-protoface	Patch
@livekit/agents-plugin-resemble	Patch
@livekit/agents-plugin-runway	Patch
@livekit/agents-plugin-sarvam	Patch
@livekit/agents-plugin-silero	Patch
@livekit/agents-plugin-soniox	Patch
@livekit/agents-plugin-tavus	Patch
@livekit/agents-plugins-test	Patch
@livekit/agents-plugin-trugen	Patch
@livekit/agents-plugin-xai	Patch

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

[CLAassistant]

Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.
_{You have signed the CLA already but the status is still pending? Let us recheck it.}