Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 9 additions & 1 deletion docs/protocol/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -40,7 +40,7 @@ msgpack maps with a `message_type` field. Arrays travel as
`dtype` is a numpy typestr: a byte-order character, a kind character, and an
item size. Parsing it takes about ten lines in any language.

## Three rules a first implementation usually gets wrong
## Four rules a first implementation usually gets wrong

**Messages strictly alternate.** `infer`, `action`, `feedback`, `infer`, and
so on. The server's connection handler is straight-line code with no
Expand All @@ -57,6 +57,14 @@ server routes feedback by environment index.
**The reward is the sum over the chunk.** Not the last step's. A client that
reports the final step's reward trains a different MDP, and nothing fails.

**A reconnect starts from nothing.** Everything the server knows about an
environment — its previous observation, the policy step state, its done
flags — lives for exactly one connection. A client that reconnects must drop
any `feedback` it was holding: the transition it describes can no longer be
completed, and sending it puts a transition built from an empty observation
into the training buffer. Nothing on either side reports an error when that
happens, which is what makes it worth stating.

## Checking an implementation

`plugrl-protocol` ships a server that grades a client against the
Expand Down
7 changes: 6 additions & 1 deletion docs/protocol/index.zh.md
Original file line number Diff line number Diff line change
Expand Up @@ -35,7 +35,7 @@ PlugRL 把一次训练拆成两个进程。**训练服务端**持有策略与学
`dtype` 是 numpy 的 typestr:一个字节序字符、一个类型字符、一个元素字节数。
用任何语言解析它大约十行代码。

## 三条最容易实现错的规则
## 四条最容易实现错的规则

**消息严格交替。** `infer`、`action`、`feedback`、`infer`……服务端的连接处理
是一段没有分发器的顺序代码,所以连发两个 `infer` 的客户端会让第二个被当成
Expand All @@ -49,6 +49,11 @@ PlugRL 把一次训练拆成两个进程。**训练服务端**持有策略与学
**奖励是整个动作块上的求和**,不是最后一步的奖励。只汇报最后一步的客户端会在
一个不同的 MDP 上训练,而且不会有任何东西报错。

**重连意味着从零开始。** 服务端关于一个环境的全部记忆 —— 上一帧观测、策略的
step state、终止标志 —— 只活在一条连接里。重连的客户端必须丢弃手上未发出的
`feedback`:它描述的那次转移已经无法补全,发出去只会往训练缓冲里塞一条由空
观测拼出来的转移。这件事发生时两端都不会报错,所以才值得单独写一条。

## 检验一个实现

`plugrl-protocol` 附带一个服务端,它按规范逐条给客户端打分,有违规就以非零码退出:
Expand Down
Loading