Skip to content

fix(realtime): the socket never dies on its own — own closes are not auth, no reconnect cap, blocked retries slowly - #6

Merged
sanctrl merged 2 commits into
mainfrom
fix/socket-never-terminal
Nov 5, 2026
Merged

sanctrl merged 2 commits into
mainfrom
fix/socket-never-terminal

Conversation

@sanctrl

@sanctrl sanctrl commented Nov 5, 2026

Copy link
Copy Markdown
Contributor

Why

A messaging client must never have a terminal state with no way out, and a close the client itself initiated is never evidence about the server. RealtimeClient broke the first rule on purpose (any 1008/4401/4403 stopped reconnecting for good) and the server uses 1008 for more than a bad key: 1008 'HELLO frame required' from its own 5 s timer (one slow key lookup or network leg with a perfectly good key), and 1008 evictions after key rotation / suspension / sign-out. Hermes and OpenClaw now receive through this socket, so it must carry the rules the OpenClaw plugin's own socket had (agentchatme/agentchat-openclaw#14), identically to the TypeScript SDK (its fix/socket-never-terminal branch; names mirrored: RealtimeState values, on('state' | 'blocked' | 'stopped'), ackNegotiated, handshakeTimeoutMs / livenessTimeoutMs, client close code 4000, initiator on the disconnect info).

State machine

idle ──connect()──▶ connecting ──hello.ok──▶ ready
   any close/open failure that is ours or not a rejection ──▶ backoff (exp ±25 %, cap 30 s; after 60 straight failures cap 5 min + one ERROR line) ──▶ connecting
   server close before hello.ok with 4401/4403 or 1008 "Invalid token" ──▶ blocked (blocked_reason, on_error, on("blocked")) ──10–15 min──▶ connecting; hello.ok clears
   watchdogs: no hello.ok within handshake_timeout_ms (15 s) / no inbound frame within liveness_timeout_ms (100 s) and no pong within 10 s ──▶ our close (4000) ──▶ backoff
   disconnect() from anywhere ──▶ stopped (only terminal state; every timer cancelled and awaited; on("stopped") last)

What changed

  • Classification (_is_auth_rejection_close): rejection = server-originated, before hello.ok, and 4401/4403 or 1008 whose reason contains invalid token (case-insensitive). Everything else, including every close this client initiated and any auth-looking code after hello.ok (eviction: the next HELLO is the evidence), takes the normal backoff.
  • Own closes go through _close_socket with code 4000 and a client ... reason; on_disconnect info gains initiator: "client" | "server".
  • No cap: max_reconnect_attempts deprecated and ignored (DeprecationWarning); long-outage cap max(max_reconnect_interval_ms, 5 min); one ERROR line at the 61st consecutive failure, reset when a connection proves stable.
  • Blocked: state == "blocked", blocked_reason ("1008 Invalid token"), slow retry 12.5 min ±20 %; first rejection logs ERROR, later DEBUG; a transport failure during the slow retry takes the fast backoff and keeps the reason; connect() while blocked retries now.
  • Watchdogs: handshake_timeout_ms (15 s, was a 4 s 1008 self-close) and liveness_timeout_ms (default 100 s on a socket with ping(), none otherwise, 0 disables). The websockets library hides pings, so liveness = inbound frames + an explicit ping probe (its pong future is never cancelled, only consumed).
  • Typed state/events: RealtimeState exported; state, blocked_reason, ack_negotiated; on("state"), on("blocked"), on("stopped"); server frames typed like these are dropped.
  • Also fixed: a HELLO send failure left an open socket with nothing reading it and no reconnect; disconnect() racing the transport open leaked the socket.
  • README + CHANGELOG (Unreleased); no version bump.

Verification (local, same gate as ci.yml)

ruff + mypy (strict) + pytest, each on CPython 3.9.6, 3.10.22, 3.11.18, 3.12.16, 3.13.18 (uv-managed venvs, websockets 15.0.1): all clean, 303 passed, 1 skipped (live smoke) per interpreter.

New tests (tests/test_realtime_resilience.py, with a controllable sleep hook and a pinging socket fake): the classification contract; 1008 'HELLO frame required' → backoff then hello.ok; 1008 'Invalid token' / invalid token / 4401 / 4403 → blocked, retry in 600–900 s, second rejection logs at DEBUG, hello.ok clears; transport failure during the slow retry → fast backoff, reason kept; connect() while blocked → at once; 75 failures → still backoff, delays ≤ 37.5 s before 60 and 225–375 s after, exactly one ERROR line; max_reconnect_attempts=1 ignored; HELLO send failure → our close + reconnect; liveness holds with pongs, trips without (code 4000, initiator: "client"), stays quiet while frames flow, not armed without ping(), 0 disables, explicit value arms frames-only; on("state") order; disconnect() from blocked and from a pending backoff → stopped, one on("stopped"), no task left in the loop. Existing tests updated: eviction after hello.ok reconnects instead of stopping; handshake timeout asserts code 4000.

Not touched: ack behaviour, frame handling, HELLO contents, the stability window, the REST drain.

…auth, no reconnect cap, blocked retries slowly

Carries the OpenClaw plugin's connection rules (agentchat-openclaw #14)
into RealtimeClient, identically to the TypeScript SDK, because Hermes and
OpenClaw now receive through the SDKs.

- A close is an auth rejection only when the server sent it before
  hello.ok and it is 4401, 4403, or 1008 with the server's auth reason
  (Invalid token). Any other 1008 (HELLO frame required from the server's
  5 s timer, an eviction after key rotation/suspension) is transient.
- A close the client initiated (handshake watchdog, liveness watchdog,
  HELLO send failure, disconnect) is never read as a verdict: code 4000
  with a `client ...` reason; on_disconnect reports initiator.
- No reconnect cap. Jittered exponential backoff up to
  max_reconnect_interval_ms; after 60 consecutive failures the cap widens
  to 5 min and one ERROR line is logged. max_reconnect_attempts is
  deprecated and ignored (warns).
- A rejection enters `blocked` (state + blocked_reason, on_error,
  on("blocked")) and retries a fresh HELLO every 10-15 min until a hello.ok
  clears it; first rejection logs at ERROR, later ones at DEBUG. An
  explicit connect() retries at once.
- Watchdogs: handshake_timeout_ms (15 s, open -> hello.ok) and
  liveness_timeout_ms (100 s without an inbound frame, then a ping probe
  with a 10 s pong deadline; default armed only on sockets with ping()).
- Typed state: RealtimeState = idle | connecting | ready | backoff |
  blocked | stopped (same names as the TypeScript SDK), on("state"),
  on("stopped"), ack_negotiated.
- Fixes a dead state on HELLO send failure and a socket leak when
  disconnect() raced the transport open.

Tests: tests/test_realtime_resilience.py (controllable sleep, pinging
socket fake); README and CHANGELOG (Unreleased) updated; no version bump.
… once per blocked run, disconnect is ours, no timer after stop, jitter below the cap

- connect() is callable from on_disconnect / on_error / on("state"): the
  guard keys on a live socket or in-flight attempt, not on the recv task
  whose finally runs those handlers; with reconnect=False a caller's own
  reconnect from on_disconnect opens a second socket instead of parking in
  idle. A handler that reconnected during a close is honoured: the old
  close arms no backoff and no blocked retry on top of it.
- on_error (and the ERROR log) fire once per blocked run; on("blocked")
  still fires per rejection, with the server's reason as sent ("" when
  none).
- disconnect() closes through _close_socket (1000 'client disconnect'), so
  on_disconnect reports initiator "client".
- _schedule_reconnect re-checks _disposed after the state event: a state
  handler that stops the client leaves no backoff timer behind.
- Non-positive handshake_timeout_ms falls back to the default.
- Backoff jitter goes up to 25 % below each step; 30 s and 5 min are true
  ceilings. The blocked retry keeps its symmetric 10-15 min range.
- Stale "4 s" comments corrected; server heartbeat facts and the
  websockets keepalive interplay noted next to the liveness default.
- Tests: connect-from-on_disconnect (reconnect on/off), disconnect from a
  state handler, 3 rejections -> 1 on_error / 3 blocked, non-positive
  handshake timeout, disconnect initiator, and one raising-on_error test
  per notice site (handshake, liveness, long outage, blocked, HELLO send).
  Frame-flow liveness test widened to a 200 ms window / 20 ms pushes.
- README and CHANGELOG (Unreleased) updated; no version bump.
@sanctrl
sanctrl merged commit 8996349 into main Nov 5, 2026
10 checks passed
sanctrl added a commit that referenced this pull request Nov 5, 2026
The v1.2.0 tag was cut after fix/socket-never-terminal (#6) merged, so its
entries belong under 1.2.0. Headings only; no content change.
@sanctrl
sanctrl deleted the fix/socket-never-terminal branch November 11, 2026 08:39
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant