From 01638324e78bfddb6d29692936e4a3d40fd7b146 Mon Sep 17 00:00:00 2001 From: Gotham-Zolio <18781106300@163.com> Date: Fri, 11 Sep 2026 14:33:07 -0400 Subject: [PATCH 1/3] docs(spec): say what a reconnect costs, in section 7.6 Section 7.2 already said the server's per-environment state goes with a closed connection, but it said it inside the resync case, as if resync were the only way a connection ends mid-run. It is not, and the consequence is the same however the connection died: the previous observation, the policy step state and the terminated/truncated flags are gone, so a feedback whose action arrived on an earlier connection describes a transition that can no longer be completed. Section 7.6 states that generally, adds the obligation on the client to drop held feedback across a reconnect, and adds the obligation on the server not to sever a healthy connection for a liveness reason section 7.3 already covers - WebSocket keepalive pings being the specific trap, since a CPU-bound learn step cannot answer one. Also a conformance checklist line, and a pointer from 7.5, which said a departing client costs nothing - true of one that is finished, not of one that means to come back. This is written up from a real failure: plugrl-server left the websockets default 20 s ping in place, and on a CPU-only machine the quickstart dropped twice in six minutes, feeding the learner an empty transition each time. Fixed in plugrl-server#4. Co-Authored-By: Claude Opus 5 (1M context) --- SPEC.md | 44 ++++++++++++++++++++++++++++++++++++++++++-- 1 file changed, 42 insertions(+), 2 deletions(-) diff --git a/SPEC.md b/SPEC.md index 1833148..f7843f6 100644 --- a/SPEC.md +++ b/SPEC.md @@ -490,8 +490,9 @@ attempting to unpack it. Everything above is the server closing. A client may also stop first - it has collected the episodes it was asked for, or its operator interrupted it - and that is not an error. The server keeps no state that outlives the -connection, so a client that disappears costs nothing beyond the feedback it -had not yet sent. +connection, so a client that is done costs nothing beyond the feedback it had +not yet sent. A client that intends to come back pays more than that; see +section 7.6. A client that is finished **SHOULD** send a WebSocket close frame with status **1000 (normal closure)** before dropping the socket, as RFC 6455 section @@ -509,6 +510,43 @@ a violation, which is the level this rule deserves. > out of steps first and closed the connection itself; E7, where the client > finishes first, made it visible on all 45 runs. +### 7.6 Reconnecting + +Section 7.2 says the server's per-environment state went with the closed +connection. That is true of **every** close, not only a resync, and it is the +one thing a reconnecting client has to reason about. + +The server holds, per environment and for exactly one connection: the +previous observation, the policy step state, and the terminated / truncated +flags. A new connection begins with none of them. So: + +* a client that reconnects **MUST** drop any `feedback` it was holding; +* it reads a fresh `metadata` and resumes from a fresh `infer`; +* a `feedback` whose `action` arrived on an earlier connection describes a + transition the server can no longer complete, because it has no previous + observation to attach it to. Sending it produces a transition built from + nothing, which is worse than the lost step it was trying to save. + +A server **SHOULD** say something when it receives feedback for an +environment it has no step state for. That condition has exactly one cause - +the connection was replaced mid-run - and storing the transition silently +puts a hole in the training data that nothing downstream can detect. + +> **The corollary for servers.** Because a reconnect costs real data, a +> server **MUST NOT** close a healthy connection for a liveness reason this +> protocol already covers. WebSocket keepalive pings are the trap: a server +> doing a CPU-bound learn step does not run its event loop, does not answer +> its own library's ping, and closes the connection under a client that is +> perfectly alive. Section 7.3's feedback timeout is this protocol's liveness +> check, and it measures the right thing - progress through the exchange +> rather than event-loop responsiveness. + +> **Historical note.** Until 2026-09-11 `plugrl-server` left the `websockets` +> default of a 20 s ping with a 20 s timeout in place. On a CPU-only machine +> the quickstart dropped its connection twice in six minutes, and each +> reconnect fed the learner one transition with an empty previous +> observation. It was invisible because nothing on either side was an error. + --- ## 8. Conformance checklist @@ -531,6 +569,8 @@ A client conforms to version 1 if it: the step that reports done; - [ ] handles close reasons `plugrl-server-stop` and `plugrl-server-resync` differently; +- [ ] drops any held `feedback` when a connection closes, and never sends + `feedback` for an `action` that arrived on an earlier connection; - [ ] treats a text frame as a fatal error. `examples/conformance_server.py` checks every clause above that is visible From 94df0f0a2b804d3861d913196f633b416e4e6aa5 Mon Sep 17 00:00:00 2001 From: Gotham-Zolio <18781106300@163.com> Date: Fri, 11 Sep 2026 14:40:58 -0400 Subject: [PATCH 2/3] docs(spec): record what turning the keepalive off gives up Section 7.6 told servers not to sever a healthy connection for a liveness reason section 7.3 already covers. It did not say that 7.3 does not cover everything, which made the advice look free. It is not free. The feedback timeout bounds the wait between an action and its feedback, and nothing bounds the wait for the next infer, so a peer that dies without closing its socket is now noticed only when TCP gives up. That is a fifth Gap and it belongs beside the other four. The alternative was a long ping timeout rather than none, and the reason it was rejected belongs in the record too: the value would have to exceed both the longest learn step and the longest legitimate pause before a client's first infer, and neither is the server's to know. Co-Authored-By: Claude Opus 5 (1M context) --- SPEC.md | 12 ++++++++++++ 1 file changed, 12 insertions(+) diff --git a/SPEC.md b/SPEC.md index f7843f6..d859101 100644 --- a/SPEC.md +++ b/SPEC.md @@ -541,6 +541,18 @@ puts a hole in the training data that nothing downstream can detect. > check, and it measures the right thing - progress through the exchange > rather than event-loop responsiveness. +> **Gap — nothing bounds the wait for the next `infer`.** Section 7.3's +> timeout covers the gap between an `action` and its `feedback`, and that is +> the only wait the server bounds. A peer that dies without closing its +> socket, between a `feedback` and the next `infer`, is noticed only when the +> operating system gives up on the TCP connection. Keepalive pings used to +> cover it, at the price above; the server now accepts the leak instead, +> because it costs one idle coroutine and a socket in a run that was going to +> stall anyway. Bounding it properly needs a number larger than the longest +> legitimate pause a client can take before its first `infer` - an +> environment reset on real hardware - and that number is not the server's to +> know. + > **Historical note.** Until 2026-09-11 `plugrl-server` left the `websockets` > default of a 20 s ping with a 20 s timeout in place. On a CPU-only machine > the quickstart dropped its connection twice in six minutes, and each From 476e1ec92c0fc32f8ecf29493c7f7c6989f7cc83 Mon Sep 17 00:00:00 2001 From: Gotham-Zolio <18781106300@163.com> Date: Fri, 11 Sep 2026 15:22:22 -0400 Subject: [PATCH 3/3] docs(spec): drop the keepalive corollary from 7.6, it was wrong The section told servers not to sever a healthy connection for a liveness reason 7.3 already covers, named WebSocket keepalive pings as the trap, and added a fifth Gap describing what turning them off gives up. All three rested on the claim that a CPU-bound learn step holds the event loop past the ping timeout. It does not. plugrl-server runs learn through asyncio.to_thread, so the loop stays free, and five learn steps of about 190 seconds each - nine times the timeout - closed nothing. The incident that prompted the section was a machine suspending for nearly two hours, and the timeout that fired was the client library's, not the server's. What is measured and survives is the rest of the section: connection state is per connection, a client must drop held feedback across a reconnect, and a server should say something when feedback arrives with no step state. Those came from reading the handler and were reproduced in a unit test, not inferred from the incident, which is why they are unaffected. The corollary and the fifth Gap are replaced by a note saying reconnects come from outside the protocol and the rule is written in terms of the reconnect for that reason. The historical note now records what actually happened, including the diagnosis that measurement overturned. Measurement in plugrl-server experiments/e8-keepalive-hypothesis. Co-Authored-By: Claude Opus 5 (1M context) --- SPEC.md | 41 +++++++++++++++-------------------------- 1 file changed, 15 insertions(+), 26 deletions(-) diff --git a/SPEC.md b/SPEC.md index d859101..ac6ba0c 100644 --- a/SPEC.md +++ b/SPEC.md @@ -532,32 +532,21 @@ environment it has no step state for. That condition has exactly one cause - the connection was replaced mid-run - and storing the transition silently puts a hole in the training data that nothing downstream can detect. -> **The corollary for servers.** Because a reconnect costs real data, a -> server **MUST NOT** close a healthy connection for a liveness reason this -> protocol already covers. WebSocket keepalive pings are the trap: a server -> doing a CPU-bound learn step does not run its event loop, does not answer -> its own library's ping, and closes the connection under a client that is -> perfectly alive. Section 7.3's feedback timeout is this protocol's liveness -> check, and it measures the right thing - progress through the exchange -> rather than event-loop responsiveness. - -> **Gap — nothing bounds the wait for the next `infer`.** Section 7.3's -> timeout covers the gap between an `action` and its `feedback`, and that is -> the only wait the server bounds. A peer that dies without closing its -> socket, between a `feedback` and the next `infer`, is noticed only when the -> operating system gives up on the TCP connection. Keepalive pings used to -> cover it, at the price above; the server now accepts the leak instead, -> because it costs one idle coroutine and a socket in a run that was going to -> stall anyway. Bounding it properly needs a number larger than the longest -> legitimate pause a client can take before its first `infer` - an -> environment reset on real hardware - and that number is not the server's to -> know. - -> **Historical note.** Until 2026-09-11 `plugrl-server` left the `websockets` -> default of a 20 s ping with a 20 s timeout in place. On a CPU-only machine -> the quickstart dropped its connection twice in six minutes, and each -> reconnect fed the learner one transition with an empty previous -> observation. It was invisible because nothing on either side was an error. +> **Where reconnects come from.** Nothing in this protocol causes them and +> nothing in it can prevent them: a suspended laptop, a flaky link, an +> operator restarting the server. The rule above is written in terms of the +> reconnect rather than its cause, because the cost is the same either way, +> and because a client cannot tell the causes apart from where it sits. + +> **Historical note.** This section exists because of a run that dropped its +> connection after the machine it was on suspended for nearly two hours. The +> reconnect was handled, and the transition that crossed it was not: the env +> client resent the `feedback` it was holding, and the server completed it +> from an empty observation and stored it. The first diagnosis blamed +> WebSocket keepalive pings and a long learn step, which measurement then +> ruled out - `plugrl-server` runs `learn` off the event loop, and learns of +> 190 s produce no ping timeout. The cause was mundane. The hole in the +> protocol was not, and had nothing to do with it. ---