From 4dc36d3bc24811aa451a6c39e83ab0d9edd3a93c Mon Sep 17 00:00:00 2001 From: Gotham-Zolio <18781106300@163.com> Date: Fri, 11 Sep 2026 14:34:30 -0400 Subject: [PATCH] docs: add the reconnect rule to the protocol tour, both languages The page listed three rules a first implementation gets wrong. There is a fourth, and it cost a real run: everything the server knows about an environment lives for exactly one connection, so a client that reconnects and sends the feedback it was holding puts a transition built from an empty observation into the training buffer. Neither side reports an error. Specified in plugrl-protocol SPEC.md section 7.6; the server-side half is plugrl-server#4. Co-Authored-By: Claude Opus 5 (1M context) --- docs/protocol/index.md | 10 +++++++++- docs/protocol/index.zh.md | 7 ++++++- 2 files changed, 15 insertions(+), 2 deletions(-) diff --git a/docs/protocol/index.md b/docs/protocol/index.md index 84efc19..7ce09dd 100644 --- a/docs/protocol/index.md +++ b/docs/protocol/index.md @@ -40,7 +40,7 @@ msgpack maps with a `message_type` field. Arrays travel as `dtype` is a numpy typestr: a byte-order character, a kind character, and an item size. Parsing it takes about ten lines in any language. -## Three rules a first implementation usually gets wrong +## Four rules a first implementation usually gets wrong **Messages strictly alternate.** `infer`, `action`, `feedback`, `infer`, and so on. The server's connection handler is straight-line code with no @@ -57,6 +57,14 @@ server routes feedback by environment index. **The reward is the sum over the chunk.** Not the last step's. A client that reports the final step's reward trains a different MDP, and nothing fails. +**A reconnect starts from nothing.** Everything the server knows about an +environment — its previous observation, the policy step state, its done +flags — lives for exactly one connection. A client that reconnects must drop +any `feedback` it was holding: the transition it describes can no longer be +completed, and sending it puts a transition built from an empty observation +into the training buffer. Nothing on either side reports an error when that +happens, which is what makes it worth stating. + ## Checking an implementation `plugrl-protocol` ships a server that grades a client against the diff --git a/docs/protocol/index.zh.md b/docs/protocol/index.zh.md index 2f2898d..58f7dc4 100644 --- a/docs/protocol/index.zh.md +++ b/docs/protocol/index.zh.md @@ -35,7 +35,7 @@ PlugRL 把一次训练拆成两个进程。**训练服务端**持有策略与学 `dtype` 是 numpy 的 typestr：一个字节序字符、一个类型字符、一个元素字节数。 用任何语言解析它大约十行代码。 -## 三条最容易实现错的规则 +## 四条最容易实现错的规则 **消息严格交替。** `infer`、`action`、`feedback`、`infer`……服务端的连接处理 是一段没有分发器的顺序代码，所以连发两个 `infer` 的客户端会让第二个被当成 @@ -49,6 +49,11 @@ PlugRL 把一次训练拆成两个进程。**训练服务端**持有策略与学 **奖励是整个动作块上的求和**，不是最后一步的奖励。只汇报最后一步的客户端会在 一个不同的 MDP 上训练，而且不会有任何东西报错。 +**重连意味着从零开始。** 服务端关于一个环境的全部记忆 —— 上一帧观测、策略的 +step state、终止标志 —— 只活在一条连接里。重连的客户端必须丢弃手上未发出的 +`feedback`：它描述的那次转移已经无法补全，发出去只会往训练缓冲里塞一条由空 +观测拼出来的转移。这件事发生时两端都不会报错，所以才值得单独写一条。 + ## 检验一个实现 `plugrl-protocol` 附带一个服务端，它按规范逐条给客户端打分，有违规就以非零码退出：