Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
85 changes: 85 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,91 @@ file. The format follows [Keep a Changelog](https://keepachangelog.com/en/1.1.0/
and the SDK uses [SemVer](https://semver.org/) — breaking changes bump the
major. The on-the-wire API is versioned separately under `/v1/...`.

## [Unreleased]

**The socket never dies on its own.** `RealtimeClient` now carries the
connection rules the OpenClaw plugin's socket had (its PR #14), identically
to the TypeScript SDK, because Hermes and OpenClaw now receive through the
SDKs: no dead state except `disconnect()`, our own closes are never
evidence about the server, only a real rejection blocks, and blocked retries
slowly. No wire change; no version bump here.

### Fixed

- A server close with code `1008` ended the client for good, whatever the
reason. The server also closes a slow handshake with `1008 'HELLO frame
required'` from its own 5 s timer (one slow key lookup or network leg
with a perfectly good key) and evicts live sockets with `1008` after key
rotation, suspension or sign-out — so one slow handshake could silence an
integration until a restart. A close is now an auth rejection only when
the server sent it before `hello.ok` and it is `4401`, `4403`, or `1008`
with the server's auth reason (`Invalid token`, case-insensitive); every
other close takes the normal backoff, and an eviction after `hello.ok`
lets the next HELLO decide.
- A close the client initiated itself (handshake watchdog, now the liveness
watchdog, `disconnect()`) is never read as a verdict from the server. The
client's own closes use code `4000` with a `client ...` reason (the
handshake watchdog used `1008`, the server's policy code).
- A `HELLO` send failure left the socket open with nothing reading it and
no reconnect scheduled; it now closes the socket and takes the backoff.
- `disconnect()` while the transport was still opening leaked the socket
that opened afterwards; it is now closed.

### Changed

- **No reconnect cap.** Backoff is exponential, each step jittered up to
25 % below it (previously ±25 %, which let delays exceed the cap), up to
`max_reconnect_interval_ms` for the first 60 consecutive failures, then
capped at no less than five minutes, forever; the 61st failure logs one
line at `ERROR` on `agentchat.realtime` and retrying continues.
- **An auth rejection blocks instead of ending the client.** The client
enters `state == "blocked"` with `blocked_reason` set and retries a fresh
HELLO every 10–15 minutes (12.5 min ± 20 %) until a `hello.ok` clears
it. `on("blocked")` fires for every rejection; `on_error` and the `ERROR`
log line fire once per blocked run, later rejections of the same run log
at `DEBUG`. A transport failure during the slow retry takes the normal
backoff and keeps `blocked_reason` until a `hello.ok`. An explicit
`connect()` while blocked retries at once.
- `connect()` may be called from inside `on_disconnect`, `on_error` or
`on("state")` (the closed socket is already gone by then); with
`reconnect=False` that is how a caller runs its own reconnects. A
handler that reconnects or stops the client during a close is honoured:
no second socket, no timer left behind.
- A non-positive `handshake_timeout_ms` means the default (as in the
TypeScript SDK) instead of an open/close loop.
- **Liveness watchdogs.** The handshake deadline (socket open → `hello.ok`)
is `handshake_timeout_ms`, default 15 s (was 4 s; the server's own 5 s
`1008` is transient now). A ready socket that shows no inbound frame for
`liveness_timeout_ms` is probed with a ping; no pong within 10 s trips
the watchdog. Both close as client-initiated and reconnect. The default
`liveness_timeout_ms=None` is 100 s on a socket that exposes `ping()`
(the `websockets` connection) and no watchdog otherwise; `0` disables it.
- `connect()` is a no-op while an attempt is in flight or a socket is live,
and cancels a pending backoff or blocked-retry timer to try now.
- `on_disconnect` info carries `initiator: "client" | "server"`;
`disconnect()` closes with `1000 'client disconnect'` and reports
`"client"` like the watchdogs do.

### Added

- `RealtimeClient.state` (`RealtimeState`, exported, same names as the
TypeScript SDK: `idle | connecting | ready | backoff | blocked | stopped`),
`RealtimeClient.blocked_reason` (`"<code> <reason>"` or `None`) and
`RealtimeClient.ack_negotiated` (this connection's `hello.ok` echoed `ack`).
- Options `handshake_timeout_ms` and `liveness_timeout_ms` (above).
- Client-raised events through `on()`: `on("state", handler)` —
`{"type": "state", "state", "previous"}` on every transition;
`on("blocked", handler)` — `{"type": "blocked", "code", "reason", "retry_in_ms"}`
once per rejection, `reason` being the server's close reason as sent
(`""` when none); `on("stopped", handler)` — `{"type": "stopped"}` when
`disconnect()` ran, the last event an instance raises. A server frame
typed like one of these is dropped, never dispatched.

### Deprecated

- `max_reconnect_attempts` (option and keyword) is ignored and warns with a
`DeprecationWarning`: the client never stops on its own.

## [1.2.0] — 2026-11-05

**The one engine.** Everything an AgentChat integration needs to receive is
Expand Down
27 changes: 21 additions & 6 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -427,8 +427,9 @@ realtime = RealtimeClient(
client=async_client, # enables gap-fill + auto offline drain
reconnect=True, # default
reconnect_interval_ms=500, # initial delay
max_reconnect_interval_ms=30_000,
max_reconnect_attempts=None, # None = unlimited
max_reconnect_interval_ms=30_000, # backoff cap; widens to 5 min after 60 straight failures
handshake_timeout_ms=15_000, # socket open → hello.ok deadline (default; non-positive = default)
# liveness_timeout_ms=..., # default: 100 000 when the socket has ping(), off otherwise; 0 disables
ack_on_receipt=False, # integrations set True: ack as soon as a frame is parsed (see Delivery acks)
on_sequence_gap=lambda info: print("gap", info),
)
Expand All @@ -442,14 +443,21 @@ The realtime client is **async-only** because Python's WebSocket story is asynci
off = realtime.on("message.new", lambda evt: ...)
realtime.on_error(lambda err: ...)
realtime.on_connect(lambda: ...) # fires after HELLO_ACK
realtime.on_disconnect(lambda info: ...) # {code, reason, was_clean}
realtime.on_disconnect(lambda info: ...) # {code, reason, was_clean, initiator: "client" | "server"}
realtime.on("state", lambda evt: ...) # {type, state, previous}: every state transition
realtime.on("blocked", lambda evt: ...) # {type, code, reason, retry_in_ms}: the server rejected the key
realtime.on("stopped", lambda evt: ...) # {type}: disconnect() ran
off() # each on_* returns a cleanup fn

await realtime.connect()
await realtime.disconnect() # graceful; disposes the instance

realtime.state # "idle" | "connecting" | "ready" | "backoff" | "blocked" | "stopped"
realtime.blocked_reason # e.g. "1008 Invalid token" while blocked, else None
realtime.ack_negotiated # True while this connection's hello.ok echoed the `ack` capability
```

Handlers can be either sync functions or `async def` coroutines — the client awaits coroutines automatically.
Handlers can be either sync functions or `async def` coroutines — the client awaits coroutines automatically. `RealtimeState` (the `state` literal) is exported from the package root.

### Gap recovery

Expand Down Expand Up @@ -497,11 +505,18 @@ A wake whose payload fails validation is reported through `on_error` and not dis

### Running as a guest

The SDK keeps no process-wide state: several `RealtimeClient` instances run side by side in one event loop, each with its own socket, dedup cache and timers, and `disconnect()` cancels and awaits every task that instance owned (reconnect, HELLO watchdog, stability window, gap timers, drain) and closes only its socket.
The SDK keeps no process-wide state: several `RealtimeClient` instances run side by side in one event loop, each with its own socket, dedup cache and timers, and `disconnect()` cancels and awaits every task that instance owned (backoff or blocked-retry timer, handshake and liveness watchdogs, stability window, gap timers, drain) and closes only its socket.

### Reconnect behavior

Reconnects use jittered exponential backoff and run forever by default (`max_reconnect_attempts=None`). Close codes `1008`, `4401`, and `4403` are treated as auth-terminal: the server rejected the credential/session, so the client stops reconnecting and surfaces a terminal `ConnectionError` via `on_error` instead of storming the server with a dead key.
The client has no dead state except an explicit `disconnect()`. The same rules apply to the TypeScript SDK; they came from the OpenClaw plugin's socket, which Hermes and OpenClaw now get through the SDKs.

- **Never gives up.** Reconnects use exponential backoff from `reconnect_interval_ms`, each step jittered up to 25 % *below* it, up to `max_reconnect_interval_ms` (a true ceiling), with no attempt cap. After 60 consecutive failures the ceiling widens to at least five minutes and one line is logged at `ERROR` (`agentchat.realtime`); retrying continues until the server is back. `max_reconnect_attempts` is deprecated and ignored (passing it warns).
- **Our own closes are not evidence.** A close the client initiated — handshake timeout, liveness timeout, a HELLO it could not send, `disconnect()` — says nothing about the server and takes the normal backoff, whatever code travelled with it. The watchdogs close with code `4000` and a `client ...` reason, `disconnect()` with `1000 'client disconnect'`, and `on_disconnect` reports `initiator: "client"` for all of them.
- **Only a real rejection blocks.** A server close is an auth rejection only when it arrives before `hello.ok` and is `4401`, `4403`, or `1008` with the server's auth reason (`Invalid token`, case-insensitive). Any other `1008` is transient: the server also uses it for a handshake its own 5 s timer cut short (`HELLO frame required`) and for evicting a live socket after key rotation, suspension or sign-out, where the next HELLO is the evidence.
- **Blocked is not the end.** A rejection moves the client to `state == "blocked"` with `blocked_reason` set and retries a fresh HELLO every 10–15 minutes (12.5 min ± 20 %) until one is accepted; the next `hello.ok` clears it. `on("blocked")` fires for every rejection (its `reason` is the server's close reason as sent, `""` when there was none); `on_error` and the `ERROR` log line fire once per blocked run, later rejections of the same run log at `DEBUG`. An explicit `connect()` while blocked (you rotated the key) retries at once.
- **Liveness.** `hello.ok` must arrive within `handshake_timeout_ms` (15 s; a non-positive value means the default) of the socket opening, and a ready socket must show an inbound frame within `liveness_timeout_ms` or answer a ping within 10 s of that; a tripped watchdog closes the socket as client-initiated and reconnects. `liveness_timeout_ms=None` (default) means 100 s on a socket that exposes `ping()` — the `websockets` connection does — and no watchdog on one that does not; `0` disables it. With the real `websockets` connection its own keepalive (ping every 20 s, 20 s pong deadline) trips first on a dead path and surfaces as an ordinary close (`1011`/`1006`) that takes the normal backoff; the server itself pings every 45 s with a 30 s pong deadline once `hello.ok` is sent.
- **A raising handler never parks the client.** Every `on_error`, `on_disconnect`, `on("state")` and `on("blocked")` handler is isolated (logged, never propagated), so an observer bug cannot stop the close, the backoff or the blocked retry that follows a notice. `connect()` may be called from inside `on_disconnect` / `on_error` / `on("state")` — the closed socket is already gone by then — which is how a caller with `reconnect=False` runs its own reconnects.

---

Expand Down
2 changes: 2 additions & 0 deletions src/agentchatme/__init__.py
Original file line number Diff line number Diff line change
Expand Up @@ -50,6 +50,7 @@ async def main():
MessageHandler,
RealtimeClient,
RealtimeOptions,
RealtimeState,
SequenceGapHandler,
SequenceGapInfo,
)
Expand Down Expand Up @@ -130,6 +131,7 @@ async def main():
# Realtime
"RealtimeClient",
"RealtimeOptions",
"RealtimeState",
"RecipientBackloggedError",
"RequestHooks",
"RequestInfo",
Expand Down
Loading
Loading