diff --git a/docs/protocol/index.md b/docs/protocol/index.md index 84efc19..7ce09dd 100644 --- a/docs/protocol/index.md +++ b/docs/protocol/index.md @@ -40,7 +40,7 @@ msgpack maps with a `message_type` field. Arrays travel as `dtype` is a numpy typestr: a byte-order character, a kind character, and an item size. Parsing it takes about ten lines in any language. -## Three rules a first implementation usually gets wrong +## Four rules a first implementation usually gets wrong **Messages strictly alternate.** `infer`, `action`, `feedback`, `infer`, and so on. The server's connection handler is straight-line code with no @@ -57,6 +57,14 @@ server routes feedback by environment index. **The reward is the sum over the chunk.** Not the last step's. A client that reports the final step's reward trains a different MDP, and nothing fails. +**A reconnect starts from nothing.** Everything the server knows about an +environment — its previous observation, the policy step state, its done +flags — lives for exactly one connection. A client that reconnects must drop +any `feedback` it was holding: the transition it describes can no longer be +completed, and sending it puts a transition built from an empty observation +into the training buffer. Nothing on either side reports an error when that +happens, which is what makes it worth stating. + ## Checking an implementation `plugrl-protocol` ships a server that grades a client against the diff --git a/docs/protocol/index.zh.md b/docs/protocol/index.zh.md index 2f2898d..58f7dc4 100644 --- a/docs/protocol/index.zh.md +++ b/docs/protocol/index.zh.md @@ -35,7 +35,7 @@ PlugRL 把一次训练拆成两个进程。**训练服务端**持有策略与学 `dtype` 是 numpy 的 typestr:一个字节序字符、一个类型字符、一个元素字节数。 用任何语言解析它大约十行代码。 -## 三条最容易实现错的规则 +## 四条最容易实现错的规则 **消息严格交替。** `infer`、`action`、`feedback`、`infer`……服务端的连接处理 是一段没有分发器的顺序代码,所以连发两个 `infer` 的客户端会让第二个被当成 @@ -49,6 +49,11 @@ PlugRL 把一次训练拆成两个进程。**训练服务端**持有策略与学 **奖励是整个动作块上的求和**,不是最后一步的奖励。只汇报最后一步的客户端会在 一个不同的 MDP 上训练,而且不会有任何东西报错。 +**重连意味着从零开始。** 服务端关于一个环境的全部记忆 —— 上一帧观测、策略的 +step state、终止标志 —— 只活在一条连接里。重连的客户端必须丢弃手上未发出的 +`feedback`:它描述的那次转移已经无法补全,发出去只会往训练缓冲里塞一条由空 +观测拼出来的转移。这件事发生时两端都不会报错,所以才值得单独写一条。 + ## 检验一个实现 `plugrl-protocol` 附带一个服务端,它按规范逐条给客户端打分,有违规就以非零码退出: