diff --git a/docs/env/libero_env.md b/docs/env/libero_env.md index 9088043..285dbc2 100644 --- a/docs/env/libero_env.md +++ b/docs/env/libero_env.md @@ -24,6 +24,41 @@ Run a few episodes. plugrl-run-env-client libero-v1 --num-episodes 10 --server-host 127.0.0.1 --server-port 8000 ``` +## As E11 ran it + +E11 evaluated a `pi05_libero` checkpoint on LIBERO through a PlugRL server: one +client process per task, ten tasks at once, with each task's initial states +taken in order rather than sampled. + +```bash +plugrl-run-env-client libero-v1 \ + --server-host 127.0.0.1 --server-port 8000 \ + --num-envs 1 --num-procs 10 --num-episodes 10 \ + --env.task-suite-name libero_spatial \ + --env.no-randomize-initial-state \ + --runner.pass-proc-id \ + --runner.replan-steps 5 \ + --runner.seed 7 \ + --recorder.no-thread0-only \ + --exp-name my_eval +``` + +- `--runner.pass-proc-id` gives process *i* task *i*, so ten processes cover a + ten-task suite and `--num-episodes` counts per task rather than in total. To + hold one task fixed instead - which is what the fine-tuning runs did - drop + that flag and pass `--env.task-id 8 --env.randomize-initial-state`. +- `--recorder.no-thread0-only` makes every process write its own + `summary.json`. Without it only process 0 reports, and per-task success + rates cannot be recovered afterwards. +- Rendering goes through EGL, so the client needs `MUJOCO_GL=egl` and + `PYOPENGL_PLATFORM=egl` in its environment. Which GPU it renders on follows + EGL's own device order, which need not agree with `CUDA_VISIBLE_DEVICES`. +- LIBERO writes its own config file, so point `LIBERO_CONFIG_PATH` somewhere + writable before the first run. + +Protocol, results and the recorded environment of both processes: +[E11](https://github.com/PlugRL/plugrl-server/tree/main/experiments/e11-vla-rl-libero). + ## Verify - Env client can create the Libero env. diff --git a/docs/env/libero_env.zh.md b/docs/env/libero_env.zh.md index 4346c0b..7185d78 100644 --- a/docs/env/libero_env.zh.md +++ b/docs/env/libero_env.zh.md @@ -24,6 +24,37 @@ plugrl-run-env-client libero-v1 --help plugrl-run-env-client libero-v1 --num-episodes 10 --server-host 127.0.0.1 --server-port 8000 ``` +## E11 是怎么跑的 + +E11 通过一个 PlugRL server 在 LIBERO 上评测 `pi05_libero` checkpoint:一个任务一个 +客户端进程,十个任务同时跑,每个任务的初始状态按顺序取,而不是随机采样。 + +```bash +plugrl-run-env-client libero-v1 \ + --server-host 127.0.0.1 --server-port 8000 \ + --num-envs 1 --num-procs 10 --num-episodes 10 \ + --env.task-suite-name libero_spatial \ + --env.no-randomize-initial-state \ + --runner.pass-proc-id \ + --runner.replan-steps 5 \ + --runner.seed 7 \ + --recorder.no-thread0-only \ + --exp-name my_eval +``` + +- `--runner.pass-proc-id` 让第 *i* 个进程跑第 *i* 个任务,十个进程正好覆盖十任务 + 套件;此时 `--num-episodes` 是**每个任务**的回合数,不是总数。若想固定跑单个 + 任务——微调那几次就是这么做的——去掉该 flag,改传 + `--env.task-id 8 --env.randomize-initial-state`。 +- `--recorder.no-thread0-only` 让每个进程都写出自己的 `summary.json`。不加它就只有 + 0 号进程上报,事后无法还原逐任务成功率。 +- 渲染走 EGL,客户端环境里需要 `MUJOCO_GL=egl` 与 `PYOPENGL_PLATFORM=egl`。它渲染 + 在哪块 GPU 上由 EGL 自己的设备顺序决定,不一定与 `CUDA_VISIBLE_DEVICES` 一致。 +- LIBERO 会写自己的配置文件,首次运行前把 `LIBERO_CONFIG_PATH` 指到可写目录。 + +协议、结果,以及两个进程各自被记录下来的环境: +[E11](https://github.com/PlugRL/plugrl-server/tree/main/experiments/e11-vla-rl-libero)。 + ## 验证 - env client 能创建 Libero 环境 diff --git a/docs/index.md b/docs/index.md index e5f0c83..6d9eacb 100644 --- a/docs/index.md +++ b/docs/index.md @@ -57,6 +57,25 @@ confirm the two sides talk to each other, not to train anything. and starts stepping episodes. - With `fpo`, the server prints a metrics table whose `rollout/reward` rises. +## A real VLA, end to end + +The same two processes carry a full-size pi0.5. The env client steps LIBERO, +the server answers with actions, and FPO trains on the feedback that comes +back. Nothing about the boundary changes; only the policy does. + +As a control, the unmodified checkpoint scored 99 of 100 on `libero_spatial` +and 185 of 200 on `libero_10`, against openpi's published 98.8 and 92.4 - and +the server's episode and step counts matched the clients' exactly, which is +what says the transport dropped nothing. + +**The reinforcement learning result is negative.** One FPO iteration on the +hardest task took its success rate from 26 of 50 to 0 of 50, and the run is +incomplete at one iteration of ten: a second learn step does not fit beside +the optimizer state the first one allocates on a 24 GB card. The predictions +were pre-registered, and one of them is falsified. The numbers, the recorded +environment of both processes, and what none of it supports: +[E11](https://github.com/PlugRL/plugrl-server/tree/main/experiments/e11-vla-rl-libero). + ## Components - `plugrl-server`: training server, runs algorithm, policy, checkpoints, tracking diff --git a/docs/index.zh.md b/docs/index.zh.md index 4ed2462..e3e84c3 100644 --- a/docs/index.zh.md +++ b/docs/index.zh.md @@ -51,6 +51,21 @@ dummy 算法的 `learn` 是一个 sleep,不会移动任何权重。它用来 - env client 打印 server 元信息(策略名、动作形状)并开始跑 episode - 用 `fpo` 时,server 的指标表里 `rollout/reward` 会上升 +## 真实 VLA,端到端 + +同样的两个进程也能承载一个完整尺寸的 pi0.5:环境端步进 LIBERO,服务端返回动作, +FPO 用回传的反馈训练。边界本身没有任何改动,变的只是策略。 + +作为对照,未经微调的 checkpoint 在 `libero_spatial` 上 99/100、在 `libero_10` 上 +185/200,与 openpi 公布的 98.8 与 92.4 一致;而且服务端记录的回合数与步数和客户端 +逐一相等——正是这一条说明传输层没有悄悄丢掉任何东西。 + +**强化学习的结果是负面的。** 在最难的那个任务上,一轮 FPO 把成功率从 26/50 打到 +0/50;而且这次训练并不完整,10 轮只跑完 1 轮——第二次 learn 装不下,它要和第一次 +分配的优化器状态挤在同一张 24 GB 卡上。预测是预注册的,其中一条被证伪。数字、两个 +进程各自被记录下来的环境,以及这些数据**不能**支持的结论,见 +[E11](https://github.com/PlugRL/plugrl-server/tree/main/experiments/e11-vla-rl-libero)。 + ## 组件 - `plugrl-server`:训练端,负责算法、策略、checkpoint、指标追踪 diff --git a/docs/policy/dppo_policy.md b/docs/policy/dppo_policy.md index 640506e..bd4c1ff 100644 --- a/docs/policy/dppo_policy.md +++ b/docs/policy/dppo_policy.md @@ -13,15 +13,28 @@ This page covers: DPPO policy. ```bash -plugrl-run-server dppo-policy default dppo hopper --exp_name my_dppo_exp +plugrl-run-server dppo-policy default dppo hopper --exp-name my_dppo_exp ``` -OpenPI PI0 policy (requires a checkpoint directory). +OpenPI PI0 policy (requires a checkpoint directory). It is a flow policy: it +pairs with `fpo` or `eval`, never with `dppo`. These are the two invocations +[E11](https://github.com/PlugRL/plugrl-server/tree/main/experiments/e11-vla-rl-libero) ran. ```bash -plugrl-run-server pi0-policy default dppo hopper \ - --policy.checkpoint_path /path/to/pi0_checkpoint \ - --policy.name pi05_tiny_libero +# evaluation - no learning +plugrl-run-server pi0-policy default eval default \ + --policy.name pi05_libero \ + --policy.checkpoint-path /path/to/pi0_checkpoint \ + --policy.device cuda + +# FPO fine-tuning +plugrl-run-server pi0-policy default fpo default \ + --policy.name pi05_libero \ + --policy.checkpoint-path /path/to/pi0_checkpoint \ + --policy.device cuda \ + --algo.learning-rate 1e-5 --algo.batch-size 8 \ + --algo.n-samples-per-action 4 --algo.buffer-size 4096 \ + --algo.global-steps 40960 ``` ## Verify @@ -52,26 +65,26 @@ python -c "import my_pkg.plugrl_policies; from plugrl_server.cli import main; ma Common flags. -- `--policy.env_type gym` -- `--policy.env_name hopper-medium-v2` -- `--policy.checkpoint_path /path/to/checkpoint.pt` +- `--policy.env-type gym` +- `--policy.env-name hopper-medium-v2` +- `--policy.checkpoint-path /path/to/checkpoint.pt` - `--policy.critic.*` ## Built-in: `pi0-policy` (OpenPI) - UID: `pi0-policy` - Code: `plugrl-server/src/plugrl_server/policy/openpi/openpi_policy.py` -- `--policy.checkpoint_path` is required and must point to a directory with: +- `--policy.checkpoint-path` is required and must point to a directory with: - `model.safetensors` - `assets/` with normalization stats - Setup notes: `plugrl-server/src/plugrl_server/policy/openpi/README.md`. Common flags. -- `--policy.name pi05_tiny_libero` -- `--policy.denoising_steps 5` -- `--policy.train_expert_only true` -- `--policy.default_prompt "..."` +- `--policy.name pi05_libero` +- `--policy.denoising-steps 5` +- `--policy.train-expert-only true` +- `--policy.default-prompt "..."` ## Implement a diffusion-style policy diff --git a/docs/policy/dppo_policy.zh.md b/docs/policy/dppo_policy.zh.md index 453f72e..9149d3d 100644 --- a/docs/policy/dppo_policy.zh.md +++ b/docs/policy/dppo_policy.zh.md @@ -13,15 +13,27 @@ DPPO 策略。 ```bash -plugrl-run-server dppo-policy default dppo hopper --exp_name my_dppo_exp +plugrl-run-server dppo-policy default dppo hopper --exp-name my_dppo_exp ``` -OpenPI PI0 策略(需要 checkpoint 目录)。 +OpenPI PI0 策略(需要 checkpoint 目录)。它是流策略:与 `fpo` 或 `eval` 搭配, +**绝不**与 `dppo` 搭配。下面两条就是 [E11](https://github.com/PlugRL/plugrl-server/tree/main/experiments/e11-vla-rl-libero) 实际跑的命令。 ```bash -plugrl-run-server pi0-policy default dppo hopper \ - --policy.checkpoint_path /path/to/pi0_checkpoint \ - --policy.name pi05_tiny_libero +# 评测 —— 不训练 +plugrl-run-server pi0-policy default eval default \ + --policy.name pi05_libero \ + --policy.checkpoint-path /path/to/pi0_checkpoint \ + --policy.device cuda + +# FPO 微调 +plugrl-run-server pi0-policy default fpo default \ + --policy.name pi05_libero \ + --policy.checkpoint-path /path/to/pi0_checkpoint \ + --policy.device cuda \ + --algo.learning-rate 1e-5 --algo.batch-size 8 \ + --algo.n-samples-per-action 4 --algo.buffer-size 4096 \ + --algo.global-steps 40960 ``` ## 验证 @@ -52,26 +64,26 @@ python -c "import my_pkg.plugrl_policies; from plugrl_server.cli import main; ma 常用参数。 -- `--policy.env_type gym` -- `--policy.env_name hopper-medium-v2` -- `--policy.checkpoint_path /path/to/checkpoint.pt` +- `--policy.env-type gym` +- `--policy.env-name hopper-medium-v2` +- `--policy.checkpoint-path /path/to/checkpoint.pt` - `--policy.critic.*` ## 内置:`pi0-policy`(OpenPI) - UID:`pi0-policy` - 代码:`plugrl-server/src/plugrl_server/policy/openpi/openpi_policy.py` -- `--policy.checkpoint_path` 必填,目录内需要: +- `--policy.checkpoint-path` 必填,目录内需要: - `model.safetensors` - `assets/`(归一化统计) - 本地安装/替换步骤见:`plugrl-server/src/plugrl_server/policy/openpi/README.md`。 常用参数。 -- `--policy.name pi05_tiny_libero` -- `--policy.denoising_steps 5` -- `--policy.train_expert_only true` -- `--policy.default_prompt "..."` +- `--policy.name pi05_libero` +- `--policy.denoising-steps 5` +- `--policy.train-expert-only true` +- `--policy.default-prompt "..."` ## 自定义 diffusion policy diff --git a/docs/policy/index.md b/docs/policy/index.md index 6f76055..c367b2e 100644 --- a/docs/policy/index.md +++ b/docs/policy/index.md @@ -24,10 +24,14 @@ plugrl-run-server --help - `fpo-policy`: FPO flow-matching policy, used by the get-started run - `pi0-policy`: OpenPI policy, requires a checkpoint path -OpenPI example. +OpenPI example. `pi0-policy` is a flow policy, so it pairs with `fpo` or with +`eval` - not with `dppo`, which expects a diffusion policy. ```bash -plugrl-run-server pi0-policy default dppo hopper --policy.checkpoint_path /path/to/checkpoint +plugrl-run-server pi0-policy default eval default \ + --policy.name pi05_libero \ + --policy.checkpoint-path /path/to/checkpoint \ + --policy.device cuda ``` ## Troubleshooting diff --git a/docs/policy/index.zh.md b/docs/policy/index.zh.md index a4bf2a1..88239f5 100644 --- a/docs/policy/index.zh.md +++ b/docs/policy/index.zh.md @@ -24,10 +24,14 @@ plugrl-run-server --help - `fpo-policy`:FPO flow matching 策略,快速上手那条命令用的就是它 - `pi0-policy`:OpenPI 策略,需要 checkpoint 路径 -OpenPI 示例。 +OpenPI 示例。`pi0-policy` 是流策略,因此与 `fpo` 或 `eval` 搭配, +**不能**与 `dppo` 搭配——后者要的是 diffusion 策略。 ```bash -plugrl-run-server pi0-policy default dppo hopper --policy.checkpoint_path /path/to/checkpoint +plugrl-run-server pi0-policy default eval default \ + --policy.name pi05_libero \ + --policy.checkpoint-path /path/to/checkpoint \ + --policy.device cuda ``` ## 常见问题