Repository navigation
feat(tts): Text-to-Speech HTTP client and WebSocket session - #200
Draft
bgotthold-aai wants to merge 25 commits into
Draft
bgotthold-aai wants to merge 25 commits into
bgotthold-aai wants to merge 25 commits into
Conversation
Adds `client.tts` for the Text-to-Speech API, matching the Python SDK: `synthesize(request)` returns one text's complete audio with the response headers, `stream(request)` returns it as the API generates it (an async iterable over a web ReadableStream), and `session(params)` opens a `TextToSpeechSession`: `connect()`, `speak()`, `generate()` / `flush()` / `audio()`, `cancel()`, `keepAlive()`, `close()`, with `on()` listeners for the server frames. Failures throw `TextToSpeechError` with the API's envelope fields. New client option `ttsBaseUrl` (US zone by default) and `createTemporaryToken()` for browser sessions. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
aleks-mitov
reviewed
Oct 23, 2026
aleks-mitov
left a comment
aleks-mitov
left a comment
There was a problem hiding this comment.
Thanks Ben, this is a big and careful port, and a lot of it is already right: request bodies and frames match the API, the error envelope maps cleanly onto TextToSpeechError, flush and cancel share one id counter (and the cancel watermark is correct for the text being written), the pull queue drops only audio on overflow, the base URL path prefix is kept, and a mid-stream cut surfaces as a retryable SDK error. It builds and lints clean and works end to end against sandbox on every runtime I tried. It is not quite ready to merge: the majors are about stopping audio and about reconnects, which are exactly the voice-agent paths this is for. Most of the HTTP ones live in TextToSpeechStream and the deadline helper, so one pass there (lock the reader when the headers arrive, keep the caller's signal attached until the body ends, cancel through the active reader, add an idle read timeout, detach listeners) covers the GC, abort, cancel, read-timeout and listener-leak comments together.
Counts: 0 blockers, 6 majors, 27 minors, 11 nits (44 inline comments).
Majors
stream()can return zero bytes with no error on Node 20 and 22 when the first read happens after an await: undici cancels the unlocked body once theResponseis garbage collected. (src/services/tts/service.ts:295)stream()stops forwardingoptions.signalonce the headers arrive, so aborting during playback does nothing. (src/services/tts/service.ts:32)TextToSpeechStream.cancel()is a silent no-op while a loop is reading (the body is locked). (src/services/tts/service.ts:200)audio()binds its flush id at the firstnext(), so an iterator created beforeflush()reads the wrong request and drops the intended one. (src/services/tts/session.ts:563)- Reconnecting from the
closelistener makes a pendingaudio()end normally, hiding the cut. (src/services/tts/session.ts:593) - A token session cannot reconnect (the token is spent or expired), but the docs and a test promise it can. (
src/services/tts/session.ts:244)
Python review fixes, carried over or not
- Carried: audio dropped before markers on overflow (but see the
enqueue()comment for the listener case), one counter for flush and cancel ids, close code from the received frame orErrorframe,Beginlimits read forgenerate(), base URL path prefix kept,containerraw by default,voice_idonly (the retiredvoiceis never sent),X-TTS-Voice-Idread, a mid-stream cut raised as the SDK error, the 120 s inactivity close in the README, reconnect after the API ends a session (with an API key). - Not yet: listener isolation, the send vs close race, token reconnects, the biometric consent errors, and on Node the long read timeout for
synthesize(). See the inline comments.
What I verified
pnpm build,lint:tsc,lint:eslint,lint:formatandlint:publintall pass, with the same output and warnings as main. Unit tests on Node 18 and 20 (the CI matrix): 385/385, including the 31 new TTS tests, which also passed 10 of 10 repeated runs with no open handles. On Node 22,pnpm test:unithas 3 failures in tests/unit/utils.test.ts that fail on main too (Node 21+ definesnavigator.userAgent), so not from this PR. typedoc adds one warning (TextToSpeechErrorFields, inline). Coverage of the new code: session.ts 88% of lines, service.ts 90%.- Every build target exports the TTS classes and
client.tts; browser.mjs and the UMD builds import nows; a strict consumer type-checks against the declarations; samples/tts.ts type-checks. Bundle growth: node.mjs from 150.9 KB to 184.7 KB, the minified UMD from 15.6 KB to 19.5 KB gzipped. - Live against sandbox with the built SDK (Node 18, 20 and 22, Bun, Deno, and the browser bundle on a WHATWG WebSocket):
synthesize()for wav, raw A-law at 8 kHz, mp3 and 44.1 kHz with every header field matching the raw response;stream()bytes equal to a raw fetch; HTTP refusals (voice_not_found, single and multi-fieldinvalid_parameter, the retiredvoiceparam, 401) mapped exactly; on sessions,Beginlimits,speak(), pipelined flushes, cancel mid-audio, flush ids after cancels matching the server's,keepAlive(), push mode, theclose()summary, an idle 3006 close followed by a reconnect with an API key, refusals beforeBegin(invalid_parameter, 1008,not_owner) with the close code from the frame, and token minting plus a token session. Against local fake servers: a body cut mid-stream gives the retryableTextToSpeechError, and the pull-queue overflow keeps theFlushDonemarkers.
Out of scope
/v1/speech/events(word timestamps over HTTP) is not wrapped, same as the Python SDK, andX-Client-Request-Id/Server-Timingare not surfaced; both look like design calls.- Not tested: a real browser engine, Cloudflare workerd, and React Native on a device (the React Native comment comes from an emulation).
- Patterns shared with
StreamingTranscriber(unguarded listeners, no handshake User-Agent) are raised only where TTS adds something new. - A few of these apply to the Python SDK too and are better fixed there separately: the flat terminate wait, the lazy
audio()binding, stale audio after a cancel,speak()past the backlog, and re-reading a finished id. Python's cancel watermark (<and>) also leaves anaudio()without an id hanging whencancel()takes its id, which this port gets right.
|
Taking this PR over to get it over the line. I'll push fixes for the review comments above, plus a few time-to-first-byte improvements for the TTS clients, as new commits on this branch (no force pushes, no history rewrites). Ben, nothing needed from you; if you have local changes that aren't pushed yet, let me know so we don't diverge. 🤖 Generated with Claude Code |
stream() now takes the response body's reader as soon as the headers arrive and hands out a stream that pulls from it. Node 20+ fetch cancels the unlocked body of a garbage-collected Response, so an iteration that started after an await could end at once with no audio and no error. TextToSpeechStream.cancel() goes through the running iteration's reader, so a barge-in callback that calls it while a for-await loop reads now ends the loop and releases the connection; before, cancelling the locked body rejected and the rejection was swallowed. Tests run the real streamed-body path for the first time: a mid-stream cut, the body locked right after stream() resolves, and cancel() from outside a running loop and before any read. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01KcyY6MAEUVjqBt5pMpetvD
…m shared signals synthesize() and stream() now share one request deadline. Its timer aborts with a DOMException named TimeoutError, so both methods time out with the same error, and stream() stops the timer once the headers arrive. The caller's signal stays attached until the request ends and is then removed, however it ends. stream() kept the caller's signal only until the headers arrived, so aborting it during playback did nothing: the body kept downloading and the loop kept yielding. Aborting it now stops the download, and the iteration throws the signal's reason (an AbortError by default) instead of the retryable cut error. synthesize() left one abort listener on the caller's signal per call. A voice agent that passes one call-level signal to every sentence's request now leaves none. The docs say that Node's built-in fetch waits at most 300 s for the headers (undici headersTimeout), so a longer synthesize() needs stream() or a dispatcher with a longer headersTimeout, and that network failures and timeouts reject with the runtime's own error. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01KcyY6MAEUVjqBt5pMpetvD
… close
- Each connection keeps its own outcome (ended, error, close) and its
`closed` item carries the error, so an audio() iteration drains its own
connection and still rejects after a close listener reconnects.
- fail() and the close handler end the connection and settle connect()
before the error listener runs, so connect() works from on("error") and
a throwing listener cannot leave a reader waiting.
- connect() rejects with the session's error marked observed, so close()
in a finally does not throw a handled refusal again; close() decides its
outcome before the close listener runs.
- The close listener gets the socket's close, or the error's close code
(1000 "Session ended" after Termination) when the socket was released
first; connect() reports a previous close that was never delivered.
- The connect timeout ends the connection and reports its close, and
audio() before the first connect() throws instead of waiting forever.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01KcyY6MAEUVjqBt5pMpetvD
…s flowing audio() was an async generator, so it picked its request at the first next(). An iterator made before generate()/flush() and iterated later (handed to a player, or to Readable.from) bound to the next request, dropped all audio of the one just flushed and waited until the session ended. audio() is now a plain method: it takes the request being written (or the given id) and the connection when called, throws a RangeError at once for an id not flushed yet, and returns the reading generator, which still checks for an audio listener or a second reader at its first next(). Readers record the highest FlushDone they consume on the connection, so audio() refuses a request already read (or skipped by reading a later one) with a RangeError instead of waiting for the session to end. A FlushDone still queued keeps its pipelined request readable. Registering an audio listener while an audio() iteration waits no longer drops the iteration's FlushDone, Cancelled and closed markers, so it ends at its FlushDone or at close(). With a listener and no iteration running, nothing is queued. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01KcyY6MAEUVjqBt5pMpetvD
…in the client's zone
A session opened with a temporary token kept the token in its readonly
params, so every reconnect resent it. The API refuses a used token where
tokens are single use and an expired one always (1008 unauthorized, not
retryable), so a browser session could not come back after a retryable
close. connect() now takes { token }: the new token replaces the
session's token for that connection and the next ones, and is sent
alone, never next to the API key.
createTemporaryToken() mints on the token host of the client's zone:
token.eu.assemblyai.com for a tts.eu.assemblyai.com ttsBaseUrl,
token.us.assemblyai.com for tts.us.assemblyai.com (the default), and
token.assemblyai.com for any other host. tokenBaseUrl still wins.
Docs: the token parameter, connect(), TextToSpeechTokenParams, the
README reconnect sentence, the CLAUDE.md gotcha, and the browser
guidance, which now opens the session with new TextToSpeechSession({
token }) (no client, no API key in the page), says to mint a token per
connection and per HTTP request, and gives the EU WebSocket URL. A
TextToSpeechSession bullet in docs/compat.md backs the session's
browser warning link.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01KcyY6MAEUVjqBt5pMpetvD
…able frame Every listener (begin, audio, flushDone, wordBoundaries, cancelled, termination, error, close) now runs through one emit() that logs a throw with console.error and lets the session go on, as the Python session's _emit does. A bug in an app's player no longer fails the session as "Unreadable frame from the API", a throwing begin listener no longer leaves connect() waiting for its timeout, and a throwing close listener no longer rejects close(). Only parsing and decoding can make a frame unreadable: a JSON message that is not an object is ignored, an Audio frame needs base64 audio and an integer flush_id, a FlushDone an integer flush_id. An unreadable frame fails the session, releases the socket so the API stops synthesizing, then runs the error and close listeners; the error listener may connect again. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01KcyY6MAEUVjqBt5pMpetvD
…connect() failure A session's handshake error now carries the HTTP status that ws reports for a refused upgrade, and is retryable only for 408, 429 and 5xx, so a 404 from a wrong websocketBaseUrl path no longer looks retryable. A browser error event with no message gets a readable one. The session constructor requires an API key or a temporary token. A socket the runtime refuses to create rejects connect() with a TextToSpeechError and clears the connect timer, and a base URL that is not ws:// or wss:// rejects connect() before anything about the previous connection changes, instead of throwing synchronously. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01KcyY6MAEUVjqBt5pMpetvD
… API drains
close() is now shared per connection: the first call sends Terminate,
and every other call on the same connection waits for that close and
resolves when the connection ends. Before, each call sent its own
Terminate and replaced the pending waiter, so a hang-up handler and a
finally that both closed the session left the first caller waiting the
whole terminateTimeout (forever with 0). Only the first call throws an
unobserved session error; later calls resolve with the summary.
terminateTimeout now counts milliseconds without a message from the API,
not time since Terminate. The wait restarts on every frame, so a
Terminate that speaks thousands of pending characters is no longer cut
at 10 s while audio is still arriving. The default is now 35,000, a
little over the API's own 30 s limit for a stalled drain, so the API's
3005 Error normally arrives first. When the wait does run out, the
connection ends with a retryable TextToSpeechError ("No Termination
within the terminate timeout; the remaining audio was dropped") on its
closed item. A running audio() iteration then rejects instead of ending
as if complete. close() still resolves undefined, session.error stays
unset, and the close listener gets 1000 "Closed by the client".
A close() that waited while a termination or error listener connected
again now resolves with its own connection's summary. Before, it ended
the new connection and closed its socket. The close listener and the
close reason now read each connection's own Termination.
Docs: close(), TextToSpeechSessionParams.terminateTimeout and the README
session paragraph.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01KcyY6MAEUVjqBt5pMpetvD
… speak() past the text backlog - Once close() has sent Terminate, generate(), flush(), cancel() and speak() throw "The session is closing: close() was called" instead of sending frames the API never reads; keepAlive() does nothing, so a keepalive timer still running during the drain cannot throw. The flag lives on the connection, so a reconnect starts clear. - A send after Begin while the socket is closing (close frame in, close event not run yet: 1006, 1011, 1001, 1012 without an Error frame) throws a retryable TextToSpeechError "The connection is closing" with the session id, not the plain "call connect() first" Error. - flush() and cancel() send first, so a refused send changes nothing: a refused cancel no longer ends the running audio() iteration. - speak() refuses text over Begin's limits.text_max_buffer_chars (when text_throttle_enabled) with a RangeError before sending anything; the API would end the session with a non-retryable 3010. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01KcyY6MAEUVjqBt5pMpetvD
…n the cancel's id The API can send audio of a request after the client's Cancel and before its Cancelled, and the API reference says to drop it. The session passed it on: an audio listener kept playing the interrupted reply, and an audio() reader waiting for its next chunk yielded the first stale frame. - dispatch() drops Audio whose flush_id is at most the last cancel's id, before the audio listener and before the pull queue. - cancel() wakes a reader waiting for its next chunk, so it ends at once instead of at Cancelled, and a reader handed a frame just before cancel() checks again before yielding it. - cancel() returns the flush_id it takes, as flush() does. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01KcyY6MAEUVjqBt5pMpetvD
A streamed body that stops sending without a reset (a dropped NAT mapping, a hung proxy) was noticed only through undici's 300 s bodyTimeout on Node, and never in a browser. stream() now takes `readTimeout` (30 000 ms by default, as in the Python SDK and the public docs; 0 turns it off): each read of the body waits at most that long, then the body is cancelled, which releases the connection, and the read rejects with the retryable cut TextToSpeechError (status 200, the request id, and the wait in the message). The timer runs only while a read waits, so playback at its own pace never trips it, and it covers a caller's own reader of `body` too. The `timeout` docs now say it covers the wait for the headers and first audio. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01KcyY6MAEUVjqBt5pMpetvD
The API records the handshake's User-Agent with a session's usage, but the Node handshake carried only Authorization, so Node SDK sessions could not be attributed. client.tts.session() now hands the client's User-Agent to the session (none with userAgent: false), and connect() sends it on Node on both the API key and the token path. A browser sends its own and its WebSocket takes no headers, so nothing changes there; a session created without a client sends none. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01KcyY6MAEUVjqBt5pMpetvD
… and document the consent errors
createTemporaryToken() refusals come from the token host as FastAPI
{"detail": ...} bodies, which speechErrorFromResponse() ignored, so a bad
key read "text-to-speech request failed with status 404". A string detail
is now the message, and a 422 list reads "<field>: <msg>" joined with "; ",
with param set to the first entry's field. The envelope's message, then
error, still win.
Document extend_biometric_consent's refusals (needs voice_id; not_owner;
biometric_terms_not_accepted until the account accepts the biometric data
terms) on both parameter types, list the cloned-voice and consent codes
with their HTTP status and close code in the errorCode doc, and add the
README sentences.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01KcyY6MAEUVjqBt5pMpetvD
…e-Id only, type bit_rate's values - SpeechResponseInfo.unknownParams lists the body fields the API ignored (X-TTS-Unknown-Params), and synthesize()/stream() warn once per process when it is set, so a camelCase `sampleRate` in a request held in a variable or sent from JavaScript shows up instead of 24 kHz audio. - voiceId comes from X-TTS-Voice-Id only: the API sends it on every 200 and no longer sends X-TTS-Voice, so the fallback that guessed a slug from it is gone. - billedCharacters and voiceExpiresAt document that on a stream they are the values as the stream started (a body cut by a server failure bills 0; a consent renewal shows on synthesize()). - bit_rate is TextToSpeechMp3BitRate (32000, 48000, 64000, 96000 or 128000; at most 64000 at 8000 Hz), as the API validates it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01KcyY6MAEUVjqBt5pMpetvD
…eStream React Native resolves the package to dist/browser.mjs, has no streamed response body and no global ReadableStream without a polyfill, so stream() downloaded (and billed) the whole audio and then threw a ReferenceError building the one-chunk fallback body. stream() now checks for a global ReadableStream first and throws a non-retryable TextToSpeechError that points to synthesize(), without sending the request. Runtimes that have ReadableStream but no streamed body keep the whole-body fallback. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01KcyY6MAEUVjqBt5pMpetvD
The type of the public TextToSpeechError constructor's second argument was
only re-exported from src/utils/errors, which the root does not export, so
`import type { TextToSpeechErrorFields } from "assemblyai"` failed with
TS2724 and a downstream declaration naming the constructor's parameters
failed with TS2742. Re-export it next to the class and name it through the
root in the #105 exports check. `pnpm generate:reference` is back to 7
warnings.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01KcyY6MAEUVjqBt5pMpetvD
…nate speaks trailing text FakeApi now answers Terminate the way the API does: text sent and not flushed is spoken first, as Audio and a FlushDone under the next flush id, before Termination. A wordBoundaries option sends each request's WordBoundaries frame after its FlushDone. New tests for behaviour that no test covered yet: - audio nobody reads keeps the newest 2,000 frames, warns once, and drops only audio, never a FlushDone, so every audio() iteration still ends - a client with a ttsBaseUrl path prefix opens sessions under that prefix - Begin's max_generate_text_length limits generate() and speak()'s frames - a close without an Error frame: 1011, 1012, 3005, 3008 and 3009 are retryable, 1008 and 3010 are not, before Begin and after it - keepAlive() sends KeepAlive; extendBiometricConsent reaches the query - close() speaks unflushed text and resolves after its FlushDone - the wordBoundaries listener receives each request's frame Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01KcyY6MAEUVjqBt5pMpetvD
…n text-to-speech in the package descriptions The README's token-streaming barge-in kept sending the LLM's tokens and then flush() after cancel(). The API reads frames in order, so that text became a new request: synthesized, billed and waited for by close(). The example now sets an interrupted flag before cancel(), stops the token loop (and the LLM stream) on it and flushes only when not interrupted. A unit test runs the example against the fake API and checks nothing follows the Cancel but Terminate. keepAlive() and inactivityTimeout now give the default (120 s), say the timer counts from the last frame in either direction, so a session can idle out while received audio still plays, and name the close (3006 inactivity_timeout, retryable). package.json's description and keywords, the README intro and CLAUDE.md mention text-to-speech, and the CLAUDE.md Key APIs entry is split into one bullet per API plus one for TextToSpeechError. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01KcyY6MAEUVjqBt5pMpetvD
The package root also exports speech-to-text types (SpeechModel, SpeechUnderstandingRequest), so the generic Speech* names read as STT. Before the first release, rename them to match the rest of the product: SpeechRequest -> TextToSpeechRequest, SpeechRequestOptions -> TextToSpeechRequestOptions, SpeechResponse -> TextToSpeechResponse and SpeechResponseInfo -> TextToSpeechResponseInfo, plus the internal header reader. The exports test names the new types through the package root and checks synthesize() and stream() are declared with them. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01KcyY6MAEUVjqBt5pMpetvD
…atency habits Audio frames decode with Node's or Bun's Buffer when one exists, read through globalThis so a browser bundle pulls in no polyfill. The bytes count only when they encode back to the frame's text exactly; any other text goes to the atob() loop, so the bytes and the refusals match atob() for every input and invalid base64 still fails the session as an unreadable frame. Chunks stay plain Uint8Arrays with an ArrayBuffer of their own. On Node 18 a 3.8 KB frame decodes in about 5 us instead of 240 us; on sandbox000 that cut the client CPU of a 6 s reply from 31.8 to 11.5 ms (p50, n=30 per arm) while text to first audio stayed the same (23.3 vs 23.1 ms). Node 20 and 22 were already fast. The TextDecoder for binary messages is made once, at the first one. README: connect the session before the reply and keep it, keepAlive() and replace it before expires_at, a spare browser token, Node fetch's 4 s idle keep-alive and undici's setGlobalDispatcher, and why Node needs no shared TLS context. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01KcyY6MAEUVjqBt5pMpetvD
…Begin The API reads frames sent before Begin once it has sent Begin, but the session refused every send until connect() resolved, so a session opened for a reply sent its text one round trip after the API's checks instead of during them. generate(), flush(), speak(), cancel() and keepAlive() called while the socket is still opening are now held and sent, in order, the moment it opens. Until Begin reports limits, the API's defaults apply (2,000 characters per Generate, 8,000 unspoken characters for speak()). A connection that fails or is closed before it opens drops the held frames; an audio() iteration then rejects with the connection's error (or ends, after close()). Measured against sandbox000 with a per-turn connect (n=30 per arm, interleaved): with 60 ms of round trip added through a delay relay, first audio came 13.8 ms sooner at p50 [95% CI -19.2, -10.3]; in region the change is within noise on Node 18/20/22, Bun and the browser bundle. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01KcyY6MAEUVjqBt5pMpetvD
…quest The ws package turns TCP_NODELAY on only once the socket is open. Under Deno's Node compatibility the upgrade request before that waits for the server's delayed ACK in most handshakes, about 40 ms. The session now passes noDelay: true to the ws socket on every runtime but the browser, whose WebSocket takes no options. Measured against sandbox000 with a per-turn connect (n=30 per arm, interleaved, in region): on Deno 2.9.7 connect to Begin went from 60.7 to 19.2 ms at p50 (-41.5 [95% CI -42.6, -40.5]) and first audio from 88.9 to 54.8 ms (-34.1 [-41.7, -16.4]); with the text sent before Begin, first audio -42.0 [-44.9, -38.7]. Node 18 and 22 and Bun (whose ws ignores the option) show no change at p50. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01KcyY6MAEUVjqBt5pMpetvD
Replaces two estimates in "Keep the time to first audio low" with measurements against sandbox000 (n=30 per arm, interleaved): - Node fetch with a 2-minute keep-alive vs Node's default, stream() requests 6 s apart: first audio -5.1 ms p50 [95% CI -9.1, -3.7] on Node 22, -9.6 [-28.9, -5.5] on Node 20 (Node 18 -4.1, CI includes 0), and -125.1 [-129.5, -120.3] with 60 ms of round trip added. Every default request opened a new TLS connection, none with keep-alive. An idle kept connection does not hold the process open; Bun and Deno fetch kept idle connections for over a minute without any setting. - A temporary token minted on the turn added 24.4 ms [22.5, 25.5] before Begin (the README said about 32 ms). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01KcyY6MAEUVjqBt5pMpetvD
stream() locks the response body at the headers so that Node 20+ fetch cannot cancel it while the caller awaits something else. That lock also turned off the runtime's release of a body nobody reads: a stream dropped without being read to its end or cancelled (a reply requested ahead that a barge-in made obsolete) kept its connection, and its listener on a shared signal, until the server gave up on the download. Before the lock, Node 20 and 22 released it at the next garbage collection. The body handed out is now registered with a FinalizationRegistry: once it is garbage-collected unread, the response body is cancelled and the request's signal released. A body that ends, however it ends, is unregistered. Reading it, iterating it or holding its reader keeps it reachable, so a stream in use is never cancelled. Checked against a local server with forced GCs: a dropped stream now releases its connection on Node 18, 20 and 22, Bun, Deno and the browser bundle (before: only on Deno), and a stream iterated, or read with the caller's own reader, through repeated GCs still delivers all its audio. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01KcyY6MAEUVjqBt5pMpetvD
Takeover summaryI pushed 24 new commits onto The six major findings, all fixed
All 27 minor and 11 nit comments are fixed as well. For five of them we took one of the routes the comment offered: the 300 s headers limit is documented instead of wrapping errors, listener exceptions are isolated, the close listener gets the close code the session already holds, Behaviour changes to know about
Time to first audioNumbers are medians measured against sandbox with light traffic, one session at a time, 30 runs per arm, with 95% confidence intervals in brackets.
Verification
Left for a human
🤖 Generated with Claude Code |
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Port of the Python SDK's Text-to-Speech clients (assemblyai-python-sdk #277 and #279) to this SDK, following its conventions: a service on the client, snake_case request bodies, camelCase session parameters, the streaming client's
on()listener pattern and WebSocket polyfill,jest-fetch-mockandjest-websocket-mocktests, a CHANGELOG entry and a minor version bump (4.42.0).What
client.tts, aTextToSpeechServiceon the newttsBaseUrl(https://tts.us.assemblyai.comby default;https://tts.eu.assemblyai.comfor the EU zone):synthesize(request)postsPOST /v1/speechbuffered and returns aSpeechResponse: the completeaudio(Uint8Array) plus the response headers asrequestId,region,model,voiceId,language,encoding,sampleRate,container,bitRate,contentType,voiceExpiresAt,billedCharacters,audioDurationMs.voiceIdreadsX-TTS-Voice-Idand, until the API sends it, is derived fromX-TTS-Voice.stream(request)returns aTextToSpeechStream: an async iterable of audio chunks over the response's webReadableStream(.body), with.infofor the headers and.bytes()to collect it. The timeout covers the headers and first audio only; leaving the loop early cancels the body; a body cut after the first audio throwsTextToSpeechErrorwithretryable: true.session(params)returns aTextToSpeechSessionfor the WebSocket API (/v1/ws):connect()waits forBegin;speak(text)returns one request's audio as an async iterable;generate()/flush()/audio(flushId?)stream text in and pipeline requests (audio()with no id follows the request being written);cancel()ends the current iteration at once;keepAlive();close()sendsTerminateand resolves with theTerminationsummary.on("begin" | "audio" | "flushDone" | "wordBoundaries" | "cancelled" | "termination" | "error" | "close", fn)for push-style use; anaudiolistener switches audio from pull to push. A session the API ended canconnect()again on the same object.createTemporaryToken({ expires_in_seconds })mints a token ontoken.assemblyai.comfor browser sessions, where the API key must not go (the same token-versus-header branching and browser warning asStreamingTranscriber).TextToSpeechErrorcarries the API's envelope:status,errorCode,param,requestId(the session id on a session),retryable,retryAfter,errors,closeCode,body. A failure nothing observed in push mode rejectsclose().src/types/tts, exported from the package root; README## Text-To-Speech, CLAUDE.md Key APIs and section,samples/tts.ts.Out of scope, as in Python: the OpenAI-compatible route,
/v1/speech/events, the Voices API (voices are managed in the dashboard).Tests
tests/unit/tts.test.ts(12) andtests/unit/tts-session.test.ts(19) againstjest-fetch-mockand a fake API overjest-websocket-mockthat answersGenerate/Flush/Cancel/Terminatethe way the server does: bodies and headers, both error shapes, retry hints, the EU host, the token path, speaking, pipelining, barge-in before and after a request, a reader following the text being written, long-text splitting on code points, push mode, lost connections, reconnecting the same object, closing during a pending connect, body cancellation on an early break, and the headers-only stream timeout.utils.test.tsuser-agent failures (they compare against the local Node version).pnpm run lint(eslint, tsc, prettier, publint) andpnpm run buildare clean.dist/node.mjs: a WAV with a RIFF header and matching duration, streamed 8 kHz mu-law, a warm session at 83 to 117 ms to first audio per sentence, barge-in with zero bytes aftercancel(), word boundaries, avoice_not_foundrefusal as a typed error on both routes.cancel(), the stream timeout covering body reads, no body cancellation on an early break, concurrent readers losing audio, a pendingconnect()outlivingclose(), UTF-16 splitting of long text, push-mode marker growth, a late pull guard inspeak(), acloselistener that never fired on client close, and an O(n) queue scan); all fixed with tests. The off-by-one also exists in the Python session and is fixed there separately.🤖 Generated with Claude Code