Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
29 changes: 29 additions & 0 deletions CLAUDE.md
Original file line number Diff line number Diff line change
Expand Up @@ -52,6 +52,11 @@ aai.settings.api_key = "your-key"
- `aai.AsyncDictationTranscriber` — Asyncio counterpart of `DictationTranscriber`; `open_live()` returns an `AsyncDictationLiveSession`. Owns an HTTP pool: use `async with` or `await aclose()`
- `aai.DictationConfig` — Dictation options: `sample_rate`, `channels`, `language_codes`, `keyterms_prompt`, `llm_instruction`, `rewrite_mode` (`aai.DictationRewriteMode.fast` default | `smart`). Rejects unknown fields
- `aai.DictationResponse` — Dictation result: `.text`, `.words` (`DictationWord` with `text` + `confidence`), `.confidence`, `.llm_response`, `.llm_error`, `.audio_duration_ms`, `.session_id`, `.request_time_ms`, `.sync_time_ms`, and `.final_text` (the LLM rewrite, falling back to `.text`)
- `aai.TextToSpeech` — Text-to-Speech HTTP API: one text per request, audio out. Methods: `synthesize()` (complete audio as a `TTSResponse`), `stream()` (a `TTSStream` context manager yielding bytes as the API generates them), `close()` / `with` for a client built from `api_key=`. Host from `aai.settings.tts_base_url` (`https://tts.us.assemblyai.com`; set `https://tts.eu.assemblyai.com` for the EU zone)
- `aai.AsyncTextToSpeech` — Asyncio counterpart of `TextToSpeech`; `synthesize()` is a coroutine and `stream()` returns an `AsyncTTSStream`. Owns an HTTP pool: use `async with` or `await aclose()`
- `aai.TTSConfig` — Text-to-Speech options: `voice_id` (a library voice such as `aai-jane`, or a cloned voice by UUID or slug), `model_id` (`assemblyai-tts-1`), `language`, `sample_rate`, `encoding` (`aai.TTSEncoding`: `pcm_s16le` default | `pcm_mulaw` | `pcm_alaw` | `mp3`), `container` (`aai.TTSContainer`: `raw` default | `wav`), `bit_rate`, `extend_biometric_consent` (owner-only consent renewal of a cloned voice). Rejects unknown fields
- `aai.TTSResponse` — Text-to-Speech result: `.audio` (bytes) plus the response headers: `.request_id`, `.region`, `.model`, `.voice_id`, `.language`, `.encoding`, `.sample_rate`, `.container`, `.bit_rate`, `.content_type`, `.voice_expires_at`, `.billed_characters`, `.audio_duration_ms`. `TTSStream.info` carries the same fields without `audio`; `audio_duration_ms` is `None` on a stream with audio (`0` when the text had nothing to speak)
- `aai.TTSError` — Raised on any Text-to-Speech refusal: `.status_code`, `.error_code` (stable API code; `biometric_terms_not_accepted` 403 until the account accepts the biometric data terms in the dashboard, `auth_unavailable` 503 retryable), `.param`, `.request_id`, `.retryable`, `.retry_after`, `.errors`, `.body` (the whole envelope)
- `assemblyai.streaming.v3.RealTimeTranscriber` — Real-time streaming with event-based API (threaded)
- `assemblyai.streaming.v3.AsyncRealTimeTranscriber` — Asyncio-native counterpart; same options/events
- `aai.LLMGateway` — LLM Gateway client. Resources: `models.list()`, `chat.completions.create()` (incl. `stream=True`), `understanding.create()`/`.validate()`
Expand Down Expand Up @@ -452,6 +457,30 @@ async with AsyncRealTimeTranscriber(RealTimeTranscriberOptions(token=token_from_
- `format_turns=True` enables punctuation/casing on confirmed end-of-turns. Toggle mid-session via `client.set_params(RealTimeSessionParameters(format_turns=True))`.
- `AsyncRealTimeTranscriber` used as `async with` calls `disconnect(terminate=True)` on normal block exit and `disconnect(terminate=False)` on exception — no explicit `disconnect()` needed inside the block.

## Text-to-Speech

One text per HTTP request. `synthesize()` returns the whole audio; `stream()` yields it as it is generated. An API refusal is always `aai.TTSError`, and so is a stream body cut after the first audio (`retryable=True`, from the loop: retry the request). Connect and first-byte failures are httpx exceptions, as in the rest of the SDK. Settings are copied when a client is built: set `tts_base_url` before creating `TextToSpeech`. `synthesize()` waits for the whole clip under `settings.tts_synthesize_http_timeout` (610 s, over the API's 600 s request cap); `stream()` uses `settings.tts_http_timeout` (30 s) per socket operation.

```python
tts = aai.TextToSpeech(config=aai.TTSConfig(voice_id="aai-jane"))

# A per-call config replaces the default whole: repeat the voice_id in it
wav = tts.synthesize("Hello.", aai.TTSConfig(voice_id="aai-jane", container=aai.TTSContainer.wav)).audio

with tts.stream("Hello.") as speech:
for chunk in speech: # raw 24 kHz s16le mono by default
play(chunk)

# Telephony: 8 kHz mu-law (chunk sizes vary; re-frame to 160 bytes for 20 ms frames)
telephony = aai.TTSConfig(voice_id="aai-jane", encoding=aai.TTSEncoding.pcm_mulaw, sample_rate=8000)
with tts.stream("Hello.", telephony) as speech:
for chunk in speech:
send_to_call(chunk)

# A cloned or library voice by id
tts.synthesize("Hello.", aai.TTSConfig(voice_id="aai-alba"))
```

## LLM Gateway

`aai.LLMGateway` is a client for AssemblyAI's LLM Gateway: model listing, OpenAI-shaped chat
Expand Down
146 changes: 146 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -890,6 +890,144 @@ except aai.DictationError as error:

---

### **Text-to-Speech Examples**

`aai.TextToSpeech` speaks one text per HTTP request on the Text-to-Speech API. `synthesize()` returns the complete audio; `stream()` returns it as the API generates it, so playback can start long before the sentence ends. A text is 1 to 5,000 characters. Options go in a `TTSConfig`: the `voice_id` (a voice of the AssemblyAI voice library by its slug: `aai-jane` by default, `aai-alba`, `aai-anna`, `aai-charles`, `aai-eve`, `aai-george`, `aai-jean`, `aai-mary`, `aai-michael`, `aai-paul`, `aai-vera` and hundreds more; or a voice cloned by your account, by its UUID or slug), the `model_id` (`assemblyai-tts-1`), the `language` (`english`, `spanish`, `german`, `french`, `italian`, `portuguese`), the `sample_rate` (8000, 16000, 22050, 24000 by default, 44100, 48000), the `encoding` (`pcm_s16le` by default, `pcm_mulaw`/`pcm_alaw` for telephony, `mp3`) and the `container` (`raw` or `wav`). A per-call `TTSConfig` replaces the client's default config whole. Voices are created and managed in the AssemblyAI dashboard.

Requests go to `aai.settings.tts_base_url`, `https://tts.us.assemblyai.com` by default; set it to `https://tts.eu.assemblyai.com` to keep them in the EU data zone.

<details>
<summary>Synthesize a WAV file</summary>

```python
import assemblyai as aai

aai.settings.api_key = "<YOUR_API_KEY>"

tts = aai.TextToSpeech()

response = tts.synthesize(
"Hello from AssemblyAI.",
aai.TTSConfig(voice_id="aai-jane", container=aai.TTSContainer.wav),
)
# A config given per call replaces the client's default config whole.

with open("hello.wav", "wb") as out:
out.write(response.audio)

print(response.model, response.voice_id, response.audio_duration_ms, "ms")
```

`synthesize()` waits for the whole audio (`aai.settings.tts_synthesize_http_timeout`, 610 s by default, just over the API's 600-second limit on a request), so an API refusal is always a `TTSError`, never a cut response, and the WAV header carries real sizes. Network failures raise httpx exceptions, as everywhere in the SDK. Python's `wave` module reads PCM WAV only; a mu-law or A-law WAV needs another reader, or use `container="raw"` for telephony codecs. The response headers are on the result: `request_id`, `region`, `model`, `voice_id`, `language`, `encoding`, `sample_rate`, `container`, `bit_rate`, `billed_characters` and `audio_duration_ms`.

</details>

<details>
<summary>Stream audio as it is generated</summary>

```python
import assemblyai as aai

aai.settings.api_key = "<YOUR_API_KEY>"

tts = aai.TextToSpeech(config=aai.TTSConfig(voice_id="aai-alba"))

with tts.stream("Streaming straight to the speakers.") as speech:
print("speaking with", speech.info.model, "at", speech.info.sample_rate, "Hz")
for chunk in speech:
play(chunk) # raw 24 kHz 16-bit mono PCM
```

`stream()` itself sends nothing: entering the `with` block (or the first read) sends the request and returns once the API has sent the first audio, usually well under a second. Chunk sizes vary and are not aligned to samples or frames: for 16-bit PCM carry an odd trailing byte over to the next chunk, and re-frame into fixed-size frames (20 ms of 8 kHz mu-law is 160 bytes) if your sink needs them. Breaking out of the loop early (a barge-in) drops that connection, so the next request pays a new TLS handshake, about 100 ms more to its first audio. The API commits to `200` with the first audio, so a failure after that cuts the body instead of changing the status: the loop then raises a `TTSError` with `retryable=True`, and the whole request should be retried.

</details>

<details>
<summary>Telephony audio and a cloned voice</summary>

```python
import assemblyai as aai

aai.settings.api_key = "<YOUR_API_KEY>"

telephony = aai.TTSConfig(
voice_id="01a0da61-138c-77c7-ab98-421431ed74e8", # a voice cloned in the dashboard
encoding=aai.TTSEncoding.pcm_mulaw,
sample_rate=8000,
)

with aai.TextToSpeech(config=telephony).stream("Your appointment is confirmed.") as speech:
for chunk in speech:
send_to_call(chunk)
```

</details>

<details>
<summary>Handle errors and retries</summary>

```python
import time
import assemblyai as aai

aai.settings.api_key = "<YOUR_API_KEY>" # or the ASSEMBLYAI_API_KEY environment variable

tts = aai.TextToSpeech()

try:
response = tts.synthesize("Hello.", aai.TTSConfig(voice_id="aai-nobody"))
except aai.TTSError as e:
print(e.status_code, e.error_code, e.param, e.request_id) # 422 voice_not_found voice_id ...
if e.retryable:
time.sleep(e.retry_after or 1)
```

`error_code` is the API's stable code (`invalid_parameter`, `invalid_request` for a voice the model does not speak, `voice_not_found`, `biometric_terms_not_accepted` (403: accept the biometric data terms in the dashboard, then retry), `rate_limited`, `at_capacity`, `auth_unavailable` (503, retryable), ...); branch on it rather than on the message. `body` holds the whole error envelope for the fields some codes add. `retryable` and `retry_after` (seconds) say whether and when to send the same request again.

</details>

<details>
<summary>Asyncio</summary>

```python
import asyncio
import assemblyai as aai

aai.settings.api_key = "<YOUR_API_KEY>"


async def main():
async with aai.AsyncTextToSpeech() as tts:
response = await tts.synthesize("Hello from AssemblyAI.")
print(len(response.audio), "bytes")

async with tts.stream("And streamed.") as speech:
async for chunk in speech:
await play(chunk) # your own coroutine


asyncio.run(main())
```

`AsyncTextToSpeech` owns an HTTP pool: use `async with` or `await aclose()`, or pass an `aai.AsyncClient` to share one.

</details>

<details>
<summary>Renew a cloned voice's consent</summary>

A voice cloned by your account expires 60 days after its biometric consent was last given. Its owner renews it by sending `extend_biometric_consent=True` with a request that uses the voice; `voice_expires_at` on the response shows the new expiry. The account must have accepted the biometric data terms in the dashboard, or the request is refused with `biometric_terms_not_accepted`.

```python
response = tts.synthesize(
"Consent renewed.",
aai.TTSConfig(voice_id="<your cloned voice's UUID>", extend_biometric_consent=True),
)
print(response.voice_expires_at) # Unix seconds
```

</details>

### **Asyncio Examples**

`aai.AsyncTranscriber` is the asyncio counterpart of `aai.Transcriber`. Every method
Expand Down Expand Up @@ -1837,8 +1975,15 @@ aai.settings.sync_live_http_timeout = 180.0
# to outlast the final segment's inference plus the LLM pass.
aai.settings.dictation_base_url = "https://dictation.assemblyai.com"
aai.settings.dictation_http_timeout = 300.0

# Text-to-Speech: the host (US by default, EU below) and the timeouts
aai.settings.tts_base_url = "https://tts.eu.assemblyai.com"
aai.settings.tts_http_timeout = 30.0 # stream(): each socket operation
aai.settings.tts_synthesize_http_timeout = 610.0 # synthesize(): the wait for the whole clip
```

Set these before creating a client: `aai.TextToSpeech()` (like every client) copies the settings when it is built, so a later change applies to the next instance, not to an existing one. Every setting can also come from the environment with the `ASSEMBLYAI_` prefix, such as `ASSEMBLYAI_TTS_BASE_URL`.

</details>

---
Expand Down Expand Up @@ -1957,6 +2102,7 @@ All custom SDK exceptions inherit from `aai.AssemblyAIError`, except the streami
| `aai.AssemblyAIError` | Base exception for all AssemblyAI errors. | `status_code: Optional[int]` |
| `aai.TranscriptError` | Raised when a prerecorded transcript API call fails at the HTTP level or polling times out. | `status_code: Optional[int]` |
| `aai.SyncTranscriptError` | Raised when synchronous transcription fails (`aai.SyncTranscriber`). | `status_code: Optional[int]`, `error_code: Optional[str]`, `retry_after: Optional[int]` |
| `aai.TTSError` | Raised when a Text-to-Speech request fails (`aai.TextToSpeech`). | `status_code: Optional[int]`, `error_code: Optional[str]`, `param: Optional[str]`, `request_id: Optional[str]`, `retryable: Optional[bool]`, `retry_after: Optional[int]`, `errors: Optional[List[Dict[str, Optional[str]]]]`, `body: Optional[dict]` |
| `assemblyai.types.RedactedAudioIncompleteError` | Raised when requesting redacted audio before processing completes. | `status_code: Optional[int]` |
| `assemblyai.types.RedactedAudioExpiredError` | Raised when requesting redacted audio that has expired and is no longer available. | `status_code: Optional[int]` |
| `assemblyai.types.RedactedAudioUnavailableError` | Raised when requesting redacted audio that is unavailable at the given URL. | `status_code: Optional[int]` |
Expand Down
22 changes: 22 additions & 0 deletions assemblyai/__init__.py
Original file line number Diff line number Diff line change
Expand Up @@ -29,6 +29,18 @@
from .prerecorded.v2 import AsyncTranscriber, AsyncTranscript
from .sync.v1 import AsyncSyncTranscriber, SyncTranscriber
from .transcriber import Transcriber, Transcript, TranscriptGroup
from .tts.v1 import (
AsyncTextToSpeech,
AsyncTTSStream,
TextToSpeech,
TTSConfig,
TTSContainer,
TTSEncoding,
TTSError,
TTSResponse,
TTSResponseInfo,
TTSStream,
)
from .types import (
AssemblyAIError,
AutohighlightResponse,
Expand Down Expand Up @@ -128,6 +140,8 @@ def __getattr__(name: str):
"AsyncDictationTranscriber",
"AsyncLLMGateway",
"AsyncSyncTranscriber",
"AsyncTTSStream",
"AsyncTextToSpeech",
"AsyncTranscriber",
"AsyncTranscript",
"AutohighlightResponse",
Expand Down Expand Up @@ -197,6 +211,14 @@ def __getattr__(name: str):
"SyncTranscriptResponse",
"SyncWord",
"Timestamp",
"TTSConfig",
"TTSContainer",
"TTSEncoding",
"TTSError",
"TTSResponse",
"TTSResponseInfo",
"TTSStream",
"TextToSpeech",
"Transcriber",
"TranscriptionConfig",
"Transcript",
Expand Down
2 changes: 1 addition & 1 deletion assemblyai/__version__.py
Original file line number Diff line number Diff line change
@@ -1 +1 @@
__version__ = "1.6.5"
__version__ = "1.7.0"
32 changes: 32 additions & 0 deletions assemblyai/tts/__init__.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,32 @@
"""The Text-to-Speech API: text in, audio out, one text per HTTP request.

The implementation lives in ``tts/v1/``, mirroring the ``dictation/v1/`` and
``sync/v1/`` layout. This ``__init__`` re-exports the public surface so
``from assemblyai.tts import TextToSpeech`` works alongside the versioned path.
"""

from .v1 import (
AsyncTextToSpeech,
AsyncTTSStream,
TextToSpeech,
TTSConfig,
TTSContainer,
TTSEncoding,
TTSError,
TTSResponse,
TTSResponseInfo,
TTSStream,
)

__all__ = [
"AsyncTTSStream",
"AsyncTextToSpeech",
"TTSConfig",
"TTSContainer",
"TTSEncoding",
"TTSError",
"TTSResponse",
"TTSResponseInfo",
"TTSStream",
"TextToSpeech",
]
25 changes: 25 additions & 0 deletions assemblyai/tts/v1/__init__.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,25 @@
from .api import TTSStream
from .async_api import AsyncTTSStream
from .async_client import AsyncTextToSpeech
from .client import TextToSpeech
from .models import (
TTSConfig,
TTSContainer,
TTSEncoding,
TTSError,
TTSResponse,
TTSResponseInfo,
)

__all__ = [
"AsyncTTSStream",
"AsyncTextToSpeech",
"TTSConfig",
"TTSContainer",
"TTSEncoding",
"TTSError",
"TTSResponse",
"TTSResponseInfo",
"TTSStream",
"TextToSpeech",
]
56 changes: 56 additions & 0 deletions assemblyai/tts/v1/_base.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,56 @@
from __future__ import annotations

from typing import Any, Dict, Optional

import httpx

from ... import _audio
from ...types import Settings
from .models import TTSConfig

__all__ = ["check_config", "check_text", "request_body", "synthesize_timeout"]


def synthesize_timeout(settings: Settings) -> httpx.Timeout:
"""
The httpx timeout of a buffered request (`synthesize()`).

Its one read waits for the whole audio, so the read timeout is
`tts_synthesize_http_timeout`; connecting and sending use
`tts_http_timeout`, as a streamed request does.
"""

return httpx.Timeout(
settings.tts_http_timeout, read=settings.tts_synthesize_http_timeout
)


def check_config(owner: str, config: Optional[TTSConfig]) -> None:
"""
Raises unless `config` is a `TTSConfig` or `None`.

Args:
owner: the class to name in the message, e.g. `TextToSpeech`.
config: the configuration to check.
"""

_audio.check_config(owner, config, TTSConfig)


def check_text(text: object) -> None:
"""Raises unless `text` is a `str`; names the swapped-argument mistake."""

if isinstance(text, TTSConfig):
raise TypeError(
"text comes first: synthesize(text, config) / stream(text, config)"
)
if not isinstance(text, str):
raise TypeError(f"text must be a str, not {type(text).__name__}")


def request_body(text: str, config: TTSConfig) -> Dict[str, Any]:
"""The JSON body of a speech request: `text` plus every option that is set."""

body: Dict[str, Any] = _audio._config_to_json(config) or {}
body["text"] = text
return body
Loading
Loading