Repository navigation
fix(realtime): the socket never dies on its own — own closes are not auth, no reconnect cap, blocked retries slowly - #6
Merged
Conversation
…auth, no reconnect cap, blocked retries slowly Carries the OpenClaw plugin's connection rules (agentchat-openclaw #14) into RealtimeClient, identically to the TypeScript SDK, because Hermes and OpenClaw now receive through the SDKs. - A close is an auth rejection only when the server sent it before hello.ok and it is 4401, 4403, or 1008 with the server's auth reason (Invalid token). Any other 1008 (HELLO frame required from the server's 5 s timer, an eviction after key rotation/suspension) is transient. - A close the client initiated (handshake watchdog, liveness watchdog, HELLO send failure, disconnect) is never read as a verdict: code 4000 with a `client ...` reason; on_disconnect reports initiator. - No reconnect cap. Jittered exponential backoff up to max_reconnect_interval_ms; after 60 consecutive failures the cap widens to 5 min and one ERROR line is logged. max_reconnect_attempts is deprecated and ignored (warns). - A rejection enters `blocked` (state + blocked_reason, on_error, on("blocked")) and retries a fresh HELLO every 10-15 min until a hello.ok clears it; first rejection logs at ERROR, later ones at DEBUG. An explicit connect() retries at once. - Watchdogs: handshake_timeout_ms (15 s, open -> hello.ok) and liveness_timeout_ms (100 s without an inbound frame, then a ping probe with a 10 s pong deadline; default armed only on sockets with ping()). - Typed state: RealtimeState = idle | connecting | ready | backoff | blocked | stopped (same names as the TypeScript SDK), on("state"), on("stopped"), ack_negotiated. - Fixes a dead state on HELLO send failure and a socket leak when disconnect() raced the transport open. Tests: tests/test_realtime_resilience.py (controllable sleep, pinging socket fake); README and CHANGELOG (Unreleased) updated; no version bump.
… once per blocked run, disconnect is ours, no timer after stop, jitter below the cap
- connect() is callable from on_disconnect / on_error / on("state"): the
guard keys on a live socket or in-flight attempt, not on the recv task
whose finally runs those handlers; with reconnect=False a caller's own
reconnect from on_disconnect opens a second socket instead of parking in
idle. A handler that reconnected during a close is honoured: the old
close arms no backoff and no blocked retry on top of it.
- on_error (and the ERROR log) fire once per blocked run; on("blocked")
still fires per rejection, with the server's reason as sent ("" when
none).
- disconnect() closes through _close_socket (1000 'client disconnect'), so
on_disconnect reports initiator "client".
- _schedule_reconnect re-checks _disposed after the state event: a state
handler that stops the client leaves no backoff timer behind.
- Non-positive handshake_timeout_ms falls back to the default.
- Backoff jitter goes up to 25 % below each step; 30 s and 5 min are true
ceilings. The blocked retry keeps its symmetric 10-15 min range.
- Stale "4 s" comments corrected; server heartbeat facts and the
websockets keepalive interplay noted next to the liveness default.
- Tests: connect-from-on_disconnect (reconnect on/off), disconnect from a
state handler, 3 rejections -> 1 on_error / 3 blocked, non-positive
handshake timeout, disconnect initiator, and one raising-on_error test
per notice site (handshake, liveness, long outage, blocked, HELLO send).
Frame-flow liveness test widened to a 200 ms window / 20 ms pushes.
- README and CHANGELOG (Unreleased) updated; no version bump.
sanctrl
added a commit
that referenced
this pull request
Nov 5, 2026
The v1.2.0 tag was cut after fix/socket-never-terminal (#6) merged, so its entries belong under 1.2.0. Headings only; no content change.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
A messaging client must never have a terminal state with no way out, and a close the client itself initiated is never evidence about the server.
RealtimeClientbroke the first rule on purpose (any1008/4401/4403stopped reconnecting for good) and the server uses1008for more than a bad key:1008 'HELLO frame required'from its own 5 s timer (one slow key lookup or network leg with a perfectly good key), and1008evictions after key rotation / suspension / sign-out. Hermes and OpenClaw now receive through this socket, so it must carry the rules the OpenClaw plugin's own socket had (agentchatme/agentchat-openclaw#14), identically to the TypeScript SDK (itsfix/socket-never-terminalbranch; names mirrored:RealtimeStatevalues,on('state' | 'blocked' | 'stopped'),ackNegotiated,handshakeTimeoutMs/livenessTimeoutMs, client close code 4000,initiatoron the disconnect info).State machine
What changed
_is_auth_rejection_close): rejection = server-originated, beforehello.ok, and4401/4403or1008whose reason containsinvalid token(case-insensitive). Everything else, including every close this client initiated and any auth-looking code afterhello.ok(eviction: the next HELLO is the evidence), takes the normal backoff._close_socketwith code4000and aclient ...reason;on_disconnectinfo gainsinitiator: "client" | "server".max_reconnect_attemptsdeprecated and ignored (DeprecationWarning); long-outage capmax(max_reconnect_interval_ms, 5 min); one ERROR line at the 61st consecutive failure, reset when a connection proves stable.state == "blocked",blocked_reason("1008 Invalid token"), slow retry 12.5 min ±20 %; first rejection logs ERROR, later DEBUG; a transport failure during the slow retry takes the fast backoff and keeps the reason;connect()while blocked retries now.handshake_timeout_ms(15 s, was a 4 s1008self-close) andliveness_timeout_ms(default 100 s on a socket withping(), none otherwise,0disables). Thewebsocketslibrary hides pings, so liveness = inbound frames + an explicit ping probe (its pong future is never cancelled, only consumed).RealtimeStateexported;state,blocked_reason,ack_negotiated;on("state"),on("blocked"),on("stopped"); server frames typed like these are dropped.disconnect()racing the transport open leaked the socket.Verification (local, same gate as
ci.yml)ruff + mypy (strict) + pytest, each on CPython 3.9.6, 3.10.22, 3.11.18, 3.12.16, 3.13.18 (uv-managed venvs, websockets 15.0.1): all clean, 303 passed, 1 skipped (live smoke) per interpreter.
New tests (
tests/test_realtime_resilience.py, with a controllable sleep hook and a pinging socket fake): the classification contract;1008 'HELLO frame required'→ backoff thenhello.ok;1008 'Invalid token'/invalid token/4401/4403→ blocked, retry in 600–900 s, second rejection logs at DEBUG,hello.okclears; transport failure during the slow retry → fast backoff, reason kept;connect()while blocked → at once; 75 failures → stillbackoff, delays ≤ 37.5 s before 60 and 225–375 s after, exactly one ERROR line;max_reconnect_attempts=1ignored; HELLO send failure → our close + reconnect; liveness holds with pongs, trips without (code 4000,initiator: "client"), stays quiet while frames flow, not armed withoutping(),0disables, explicit value arms frames-only;on("state")order;disconnect()from blocked and from a pending backoff →stopped, oneon("stopped"), no task left in the loop. Existing tests updated: eviction afterhello.okreconnects instead of stopping; handshake timeout asserts code 4000.Not touched: ack behaviour, frame handling, HELLO contents, the stability window, the REST drain.