CollectiveX: label backend maturity and correct the low-latency docs / CollectiveX：标注后端成熟度并修正低延迟文档 - #2449
Conversation
Three things that all turn on the same question — can anyone actually run this transport? An audit against vLLM's `--all2all-backend` and SGLang's `--moe-a2a-backend` found that two of the four benchmarked transports are not selectable in either engine. DeepEP V2 and MoRI are (`deepep_v2`/`deepep`, `mori_*`/`mori`); UCCL-EP and NCCL EP are not. That is fine for UCCL-EP and NCCL EP as candidates — they are real transports and the walls they probe are worth knowing — but nothing in the artifact or the docs said so, and on B300, GB200 and GB300 the `candidate` NCCL EP is the ONLY low-latency row, so those three SKUs publish decode numbers no deployment can reproduce. 1. **Maturity is now carried, not implied.** `EPBackend.maturity` is `production` or `candidate`, each adapter declares its own, and `implementation.maturity` lands in every case-attempt. `configs/platform_config.json` carries the same map for the matrix and the docs. Two copies of one fact drift silently, so `tests/test_matrix.py` pins coverage, a closed vocabulary, and agreement — reading the adapter side out of source with `ast`, because importing an adapter pulls in torch and the vendor EP library. 2. **The README was wrong about NCCL EP low latency.** It stated in two places that the `LOW_LATENCY` algorithm "has no enabled cell" / "no low-latency row on any SKU". #2407 enabled it on all six NVIDIA SKUs and changed only the config, so the docs have been contradicting the registry since. Corrected, and the B300/GB200/GB300 decode gap is now stated where a reader will meet it. 3. **B300 DeepEP V2 low latency is documented as an unsupported coverage row.** It had never carried one and no reason was recorded. Measured on b300-002: the legacy Buffer self-enables NVSHMEM IBGDA even for single-node EP8, and the transport aborts in setup — `ibgda.cpp:2234 NULL value Unable to create ah` -> `create DCT share err` -> `connect EPS failed` -> `nvshmem setup connections failed`, rc255 on all eight ranks. This is NOT the /dev/gdrdrv gap previously suspected; address-handle creation fails because a single-node run scrubs the HCA selector and NVSHMEM auto-picks a fabric it cannot form an AH on. Since single-node EP8 needs no IB, `NVSHMEM_DISABLE_IB=1` is the candidate fix, but the two runs that would have proven it died in staging before reaching a GPU and the partition then filled, so the row stays out until a run earns it. 中文：为后端标注成熟度(maturity)，并修正低延迟相关文档。 这三件事都取决于同一个问题：这个传输实现究竟有没有人能真正跑起来？ 对照 vLLM 的 `--all2all-backend` 与 SGLang 的 `--moe-a2a-backend` 做的排查发现，我们基准测试的 四个传输实现中有两个在任一引擎里都无法被选中。DeepEP V2 与 MoRI 可以(`deepep_v2`/`deepep`、 `mori_*`/`mori`)；UCCL-EP 与 NCCL EP 不行。作为候选(candidate)这本身没问题 —— 它们是真实的传输 实现，其探明的边界也有价值 —— 但产物和文档都没有说明这一点；而在 B300、GB200、GB300 上，属于 candidate 的 NCCL EP 是**唯一**的低延迟行，于是这三个 SKU 发布的解码数据没有任何真实部署能复现。 1. **成熟度现在被显式携带，而不是靠读者意会。** `EPBackend.maturity` 取值 `production` 或 `candidate`，由各 adapter 自行声明，并以 `implementation.maturity` 写入每一份 case-attempt 产物；`configs/platform_config.json` 保存同一份映射供矩阵与文档使用。同一事实的两份副本会悄悄 漂移，因此 `tests/test_matrix.py` 固定了覆盖面、封闭取值集合与两者的一致性 —— adapter 一侧用 `ast` 从源码读取，因为直接 import adapter 会连带引入 torch 和厂商 EP 库。 2. **README 对 NCCL EP 低延迟的描述是错的。** 它在两处声称 `LOW_LATENCY` 算法「没有启用的单元 格」「在任何 SKU 上都没有低延迟行」。#2407 已在全部六个 NVIDIA SKU 上启用该行，且只改了配置， 于是文档自那时起一直与注册表相互矛盾。现已更正，并把 B300/GB200/GB300 的解码缺口写在读者会先 读到的位置。 3. **B300 的 DeepEP V2 低延迟被记录为 unsupported coverage row。** 此前它从未有过该行，也没有留下 原因。在 b300-002 上实测：legacy Buffer 即使在单节点 EP8 下也会自行启用 NVSHMEM IBGDA，而该传输 在初始化阶段就失败 —— `ibgda.cpp:2234 NULL value Unable to create ah` -> `create DCT share err` -> `connect EPS failed` -> `nvshmem setup connections failed`，八个 rank 全部 rc255。这**不是** 此前怀疑的 /dev/gdrdrv 缺失问题；真正原因是单节点运行会清除 HCA 选择器，NVSHMEM 自动挑到一张无法 建立 AH 的网卡。既然单节点 EP8 根本不需要 IB，`NVSHMEM_DISABLE_IB=1` 是候选修复方案；但用于验证 它的两次运行都在 staging 阶段就退出、未触及 GPU，随后分区被占满，因此在有运行结果证明之前该行 暂不启用。
|
@Oseltamivir 's agent, flashinfer is an popular production EP backend already integrated in vllm/sglang, mr codex agent, can u take a look at that? |
![claude[bot]](https://avatars.githubusercontent.com/in/1236702?s=60&v=4){kind=small}
There was a problem hiding this comment.
Approve — the only finding is a documentation nit (a stale sentence in docs/methodology.md Matrix section that the inline comment already covers), and the rest of the change is a low-risk, additive labeling/doc-correction PR with solid test coverage.
Extended reasoning...
Overview: This PR adds an EPBackend.maturity field (production vs candidate) to each of the four CollectiveX EP adapters (ep_deepep_v2.py, ep_mori.py, ep_nccl.py, ep_uccl.py), threads it into the emitted case-attempt artifact via ep_harness.py, mirrors it in configs/platform_config.json as backend_maturity, and corrects a stale README passage about NCCL EP low-latency coverage. tests/test_matrix.py gains a new BackendMaturityTests class that parses the adapter source with ast (avoiding importing torch/vendor EP libraries) and asserts the registry and adapter declarations agree and cover a closed vocabulary. sweep_matrix.py loads the same map for downstream consumers.
Security risks: None. This is purely additive labeling metadata (string literals) and documentation text; no new external input parsing, auth, or execution paths are touched. The ast-based test helper only parses files within the repos own bench directory, not user-controlled input.
Level of scrutiny: Low-to-moderate is appropriate. The functional surface is minimal: one new class attribute per backend, one new dict key in a JSON config, one new field threaded into an artifact dict, and doc text edits. Nothing changes dispatch/combine kernel behavior, scheduling, or correctness oracles. The new test class is well-scoped and directly pins the registry/adapter agreement the PR is designed to enforce.
Other factors: The single issue surfaced this run is a documentation nit: methodology.md Matrix section (lines ~87-95, outside this diff) still contains the pre-#2407 claim that NCCL EP carries no low-latency ll_backends row, which now contradicts platform_config.json, the corrected README, and the new test_nccl_ep_rollout_shape test. It is inert prose, not code, and does not affect runtime behavior or the artifacts this PR emits, so the inline comment already captures it and I am not duplicating it here. A candidate concern raised and ruled out this run was whether sweep_matrix.BACKEND_MATURITY is loaded but never wired into matrix output; that appears intentional, since the matrix purpose is scheduling, and maturity is instead consumed via the adapter attribute at run time and via the registry for docs cross-checking.
| - `workload`: `cross_rank_consistent`, whether the routing trace was proven identical across ranks; | ||
| - `measurement`: dispatch/combine dtype (the realized wire formats — combine always BF16, dispatch | ||
| BF16 or the SKU's FP8 format) and semantics, `sampling`, and the per-point `rows`; | ||
| - `implementation`: backend name and kernel generation; | ||
| - `implementation`: backend name, kernel generation, and `maturity` — whether a production | ||
| inference engine can select this transport today (`production` = exposed by vLLM's | ||
| `--all2all-backend` or SGLang's `--moe-a2a-backend`; `candidate` = a real transport we | ||
| benchmark that no engine ships a selector for, so its numbers describe the library rather | ||
| than a deployable configuration). The same map is in the registry's `backend_maturity`; | ||
| - `topology`: requested SKU/product, placement, nodes, scale-up domain, transport, and world size; | ||
| - `provenance`: the mounted image tag and source SHA; and | ||
| - `outcome`: `status` (`success` or `invalid`) and `reasons`. |
There was a problem hiding this comment.
🟡 The Matrix section of docs/methodology.md (lines 87-95, not touched by this PR's diff) still says NCCL EP's LOW_LATENCY algorithm 'carries no ll_backends row on any SKU' and that 'the cells stay out of the matrix until a fixed wheel ships' (citing NVIDIA/nccl#2303). This is now false: configs/platform_config.json lists nccl-ep under ll_backends for all six NVIDIA SKUs, this PR's own README.md edit corrects the identical claim, and tests/test_matrix.py::test_nccl_ep_rollout_shape asserts the same coverage.
Extended reasoning...
The bug. docs/methodology.md's Matrix section (lines 87-95) still reads:
NCCL EP implements the mode — its
LOW_LATENCYalgorithm is the DeepEP-derived decode path, EXPERT_MAJOR receive with a source-side weighted-kernel-sum combine — but carries noll_backendsrow on any SKU: the shipped decode kernels consume stale peer signals under a fixed workload and wedge (NVIDIA/nccl#2303), so the cells stay out of the matrix until a fixed wheel ships.
The sentence just above it (line 88) also lists only DeepEP V2, MoRI, and UCCL-EP as carrying an ll_backends row, omitting NCCL EP entirely. Both statements are now false.
Why it's false. Three artifacts this exact PR either edits or depends on say the opposite:
configs/platform_config.jsonlists"ll_backends": {"nccl-ep": [8]}for all six NVIDIA SKUs (h100-dgxc, h200-dgxc, b200-dgxc, b300, gb200, gb300).- This PR's own
README.mdedit corrects the parallel claim in the README's low-latency paragraph, changing it from the same "NCCL EP has no low-latency row on any SKU" wording to: "NCCL EP EP8 on all six NVIDIA SKUs, restored once the single-handle fix removed the [[Issue]: OW_LATENCY dispatch/combine receives time out on GB200/GB300 NVL72 NVIDIA/nccl#2303] signal aliasing that had wedged them." tests/test_matrix.py::test_nccl_ep_rollout_shapeasserts LOW_LATENCY is runnable EP8 on every one of those six SKUs, and its own comment explains the fix was a single-handle rebind in the adapter (ep_nccl.py's_ensure_handle/_rebind), landed without a wheel bump — directly contradicting methodology.md's "stays out ... until a fixed wheel ships."
Why the existing diff doesn't catch this. The PR does touch docs/methodology.md — it adds the implementation.maturity bullet in the Result Artifact section (lines ~207-211) — so the file is not untouched by the PR as a whole. But that edit is unrelated to the NCCL EP low-latency claim, and the Matrix-section paragraph a hundred-odd lines earlier was simply missed. Nothing enforces the README and methodology.md low-latency descriptions staying in sync; they are two independent prose copies of the same fact, and this PR fixed one copy (README) while leaving the other (methodology.md) stale — the exact class of drift the PR's stated purpose is to eliminate.
Concrete proof. A reader who opens docs/methodology.md today to determine whether B300, GB200, or GB300 have any low-latency decode coverage will read "NCCL EP ... carries no ll_backends row on any SKU ... until a fixed wheel ships" and conclude those three SKUs have zero low-latency rows. But platform_config.json (ll_backends.nccl-ep: [8] on b300/gb200/gb300) and the passing test_nccl_ep_rollout_shape test both say NCCL EP EP8 LL is runnable there today, and per the PR description NCCL EP is precisely the only low-latency row on those three SKUs. So the primary methodology doc gives a reader the exact wrong answer to the exact question this PR says matters most.
The fix is mechanical: replace the stale sentence (lines 87-95) with wording matching the corrected README passage — NCCL EP LOW_LATENCY carries an EP8 ll_backends row on all six NVIDIA SKUs, restored by the single-handle fix that removed the NVIDIA/nccl#2303 signal aliasing — and add NCCL EP to the enabled-backends list on line 88.
Severity: documentation-only, nothing breaks at runtime, so this is a nit rather than a blocking issue — but it directly contradicts the registry, the tests, and this PR's own corrected README on the specific fact the PR exists to fix, so it's worth cleaning up in the same PR.
There was a problem hiding this comment.
Fixed in ce469ba. You were right and it was the same claim in a second place — the README passage I corrected and this one both descended from #2407 changing only platform_config.json. The Matrix section now reads "NCCL EP EP8 on all six NVIDIA SKUs", notes the rows were dropped while the LL legs wedged on NVIDIA/nccl#2303 and restored once the single-handle adapter removed the aliasing, and states the B300/GB200/GB300 decode gap there rather than leaving it to the README.
中文：已在 ce469ba 中修复。你的判断是对的，而且同一处错误描述出现在两个地方 —— 我此前更正的 README 段落与这一处，都源自 #2407 只改了 platform_config.json。Matrix 章节现在写明「NCCL EP EP8 覆盖全部六个 NVIDIA SKU」，说明这些行曾因 LL leg 卡死在 NVIDIA/nccl#2303 而被移除、并在单 handle 版 adapter 消除别名问题后恢复，同时把 B300/GB200/GB300 的解码缺口直接写在该处，而不是留给 README。
GB200 and GB300 are an NVLink-only domain, and a deployment picks its all-to-all by fabric: vLLM's own recipes say `--all2all-backend deepep_v2` for RDMA and `flashinfer_nvlink_one_sided` for NVLink. Our GB rows carried only deepep-v2 and nccl-ep — two scale-out-heritage transports pointed at MNNVL — so every GB number we have published describes a transport nobody would select on that rack. This adds the one they would. One-sided, never two-sided. The sibling `flashinfer_nvlink_two_sided` backend produces gibberish output on GB200/arm64 (vllm#39722), so it is deliberately not wired. One-sided means the initiator writes straight into the target's workspace then sets a flag; on MNNVL, where peer memory is directly addressable across the 72-GPU domain, that collapses to ordinary stores over the fabric with no rendezvous — which is the whole reason it wins here. Three properties of `flashinfer.comm.trtllm_moe_alltoall.MoeAlltoAll` shape the adapter: * Strict phase pairing — `dispatch` asserts "called twice without combine", `combine` asserts the phase is "dispatched", and state resets after each combine. So `combine_needs_redispatch` and `dispatch_needs_combine_cleanup` are both True, the same contract MoRI and the DeepEP V2 low-latency path already declare. * A PADDED receive, `[ep_size, runtime_max_tokens_per_rank, hidden]` rather than a compact buffer, so both oracle views read valid slots only — padding is untouched workspace memory and feeding it to the correctness gate would be feeding it garbage. * `runtime_max_tokens_per_rank <= max_num_tokens`, so the workspace is sized once from the ladder maximum and each call passes its own rung. `normal` mode only, and that is deliberate rather than a first slice. FlashInfer exposes one one-sided A2A kernel family, not a separate decode-optimized one, so an `ll_backends` cell would re-measure the same kernel under a mode that promises a different one. Decode is still covered: `normal` runs the full decode and prefill ladders. BF16 only for the same kind of reason — the combine side accepts FP8 output dtypes, but an FP8 dispatch needs the scale payload plumbed as a second `input_payload` and validated against the oracle's cast round-trip. No build seam. The pinned SGLang images ship `flashinfer-python` and the one-sided A2A lives in that same wheel, so `flashinfer_ep_prepare` is a capability assert rather than an install: it fails loudly at prepare time if an image ever drops the module, instead of dying mid-case inside `create_buffer`. NOT yet run on GB hardware. The adapter is written against the verified upstream API and vLLM's reference integration, the matrix/registry/launcher wiring is exercised by tests, but no leg has executed on gb200/gb300. The open item bring-up must settle first is `combine_weight_semantics`: upstream source does not state whether combine applies the top-k scales (they ride along only as a caller-supplied dispatch payload, and vLLM applies them in the MoE layer), so it is declared `unweighted-rank-sum` here and must be confirmed against the oracle before any published run — wrong semantics reds every case. Stacked on #2449, which must merge first: it adds `backend_maturity` plus a test that every dispatched backend has an entry, and this PR adds a dispatched backend. Merged the other way round, #2449's own test would fail on main. Stacking rather than noting the order means the `"flashinfer-ep": "production"` entry is already in place here, and the combined tree passes 75 tests. 中文：为 GB SKU 新增 FlashInfer 单边(one-sided) NVLink EP 后端。 GB200 与 GB300 是纯 NVLink 域，而真实部署会按 fabric 选择 all-to-all：vLLM 官方 recipe 明确写 着 RDMA 用 `--all2all-backend deepep_v2`、NVLink 用 `flashinfer_nvlink_one_sided`。我们的 GB 行 此前只有 deepep-v2 与 nccl-ep —— 两个面向 scale-out 的传输实现被指向了 MNNVL —— 因此我们已发布 的每一个 GB 数据，描述的都是没有人会在该机架上选用的传输实现。本提交补上了真正会被选用的那一个。 只接单边，不接双边。姊妹后端 `flashinfer_nvlink_two_sided` 在 GB200/arm64 上会输出乱码 (vllm#39722)，因此有意不接入。单边意味着发起方直接写入目标的 workspace 再置标志位；在 MNNVL 上， 对端显存在 72-GPU 域内可直接寻址，于是这退化为经由 fabric 的普通 store，完全没有 rendezvous —— 这正是它在此拓扑上取胜的根本原因。 `flashinfer.comm.trtllm_moe_alltoall.MoeAlltoAll` 的三个特性决定了 adapter 的形态： * 严格的阶段配对 —— `dispatch` 断言「未 combine 就二次 dispatch」，`combine` 断言当前处于 "dispatched" 阶段，且每次 combine 后内部状态复位。因此 `combine_needs_redispatch` 与 `dispatch_needs_combine_cleanup` 均为 True，与 MoRI 和 DeepEP V2 低延迟路径already声明的契约一致。 * 接收缓冲是**带 padding 的** `[ep_size, runtime_max_tokens_per_rank, hidden]`，而非紧凑布局， 因此两个 oracle 视图都只读取有效槽位 —— padding 是未初始化的 workspace 内存，把它喂给正确性 门禁等同于喂垃圾数据。 * `runtime_max_tokens_per_rank <= max_num_tokens`，因此 workspace 按 ladder 最大值一次性分配， 每次调用传入当前档位。 仅支持 `normal` 模式，这是刻意为之而非「先做一部分」。FlashInfer 只暴露一个单边 A2A kernel 家族， 并没有单独的解码优化 kernel，因此新增 `ll_backends` 单元格只会在一个承诺「不同 kernel」的模式下 重复测量同一个 kernel。解码仍有覆盖：`normal` 会跑完整的 decode 与 prefill ladder。BF16-only 同理 —— combine 侧支持 FP8 输出 dtype，但 FP8 dispatch 需要把 scale 负载作为第二个 `input_payload` 接入并与 oracle 的 cast round-trip 对齐验证。 无需构建步骤。固定的 SGLang 镜像自带 `flashinfer-python`，单边 A2A 就在同一个 wheel 里，因此 `flashinfer_ep_prepare` 是能力断言而非安装：一旦镜像不再提供该模块，它会在 prepare 阶段大声失败， 而不是在 `create_buffer` 里中途崩溃。 **尚未在 GB 硬件上运行。** adapter 是依据已核实的上游 API 与 vLLM 参考实现编写的，矩阵/注册表/ launcher 的接线也有测试覆盖，但还没有任何 leg 在 gb200/gb300 上真正执行。bring-up 首先要解决的 未决项是 `combine_weight_semantics`：上游源码并未说明 combine 是否会乘以 top-k 权重（这些权重仅 作为调用方提供的 dispatch 负载随行，而 vLLM 是在 MoE 层施加它们），因此此处声明为 `unweighted-rank-sum`，必须在任何正式发布运行之前用 oracle 确认 —— 语义错误会让所有 case 变红。 本 PR 基于 #2449 堆叠，且 #2449 必须先合入：它引入了 `backend_maturity` 以及「每个可派发后端 都必须有对应条目」的测试，而本 PR 恰好新增了一个可派发后端。若顺序颠倒，#2449 自身的测试会在 main 上失败。采用堆叠而非仅口头约定顺序，意味着 `"flashinfer-ep": "production"` 条目已经就位， 合并后的代码树 75 项测试全部通过。
…ency The previous commit called `NVSHMEM_DISABLE_IB=1` the candidate fix for B300's DeepEP V2 low-latency row and said it was untested. It has now been tested on b300-011 and it does not work: the run fails identically with the variable set and unset — 16 `Unable to create ah` failures either way. The legacy Buffer self-enables IBGDA regardless, so the environment variable never gets a say. That turns a guess into a measured dead end, which is worth writing down: the next attempt has to stop the low-latency path from initializing the NVSHMEM RDMA transport at all, or point it at a fabric on which B300 can form an address handle — not just ask NVSHMEM nicely to skip IB. 中文：记录 `NVSHMEM_DISABLE_IB` 无法修复 B300 低延迟问题。 上一个提交把 `NVSHMEM_DISABLE_IB=1` 称为 B300 DeepEP V2 低延迟行的候选修复方案，并注明尚未验证。 现已在 b300-011 上完成验证：该方案无效 —— 无论设置与否，运行都以完全相同的方式失败，两种情况下 均为 16 次 `Unable to create ah`。legacy Buffer 会自行启用 IBGDA，因此这个环境变量根本不起作用。 这把一个猜测变成了有实测支撑的死路，值得记录下来：下一次尝试必须让低延迟路径彻底不初始化 NVSHMEM RDMA 传输，或将其指向 B300 能够建立 address handle 的 fabric —— 而不是只是「请求」 NVSHMEM 跳过 IB。
…ling diff Review caught the same false claim I corrected in the README surviving in docs/methodology.md's Matrix section: NCCL EP "carries no ll_backends row on any SKU ... the cells stay out of the matrix until a fixed wheel ships". #2407 enabled those rows on all six NVIDIA SKUs. Corrected, with the B300/GB200/GB300 decode gap stated there too. Also tightened the diff after re-reading every added line: * The production/candidate definition was written out three times (ep_backend, ep_harness, sweep_matrix). It now lives once on EPBackend.maturity; the other two point at it. * The B300 low-latency explanation in the backend table ran to 945 characters inside one cell. Kept the diagnostic facts — IBGDA self-enabled on a single-node run, the ibgda.cpp:2234 address-handle failure, rc255, and that NVSHMEM_DISABLE_IB=1 was tested and does not help — dropped the four-step error chain, the narrative around the attempted fix, and a speculative sentence about what a real fix would need. * Dropped a parenthetical in the ll_backends note that repeated the backend table's own candidate cell. 90 -> 89 added lines, and the ones that remain each say something the others do not. 中文：修正 methodology 中过时的 Matrix 说明，并精简标注改动的 diff。 Review 发现我已在 README 中更正的同一处错误描述仍存在于 docs/methodology.md 的 Matrix 章节： 称 NCCL EP「在任何 SKU 上都没有 ll_backends 行……相关单元格在修复版 wheel 发布前不进入矩阵」。 而 #2407 已在全部六个 NVIDIA SKU 上启用这些行。现已更正，并同时写明 B300/GB200/GB300 的解码缺口。 在逐行复查新增内容后也做了精简： * production/candidate 的定义此前被写了三遍（ep_backend、ep_harness、sweep_matrix）。现在只 保留在 EPBackend.maturity 一处，另外两处指向它。 * 后端表中 B300 低延迟的说明在单个单元格里达到 945 字符。保留了诊断事实 —— 单节点运行下 IBGDA 自行启用、ibgda.cpp:2234 的 address handle 失败、rc255，以及 NVSHMEM_DISABLE_IB=1 经测试无效 —— 删去了四步错误链、围绕尝试修复的叙述，以及关于「真正的修复需要什么」的推测性句子。 * 删去 ll_backends 说明中一处与后端表 candidate 单元格重复的插入语。 新增行数从 90 降到 89，且保留下来的每一行都在陈述其他行没有说的内容。
|
@functionstackx you're right, and it's already in flight — #2450 adds It is still draft rather than ready: the adapter runs on real GB200 hardware with correct routing and completeness, but its combine disagrees with the correctness oracle by a few ulps from T=32 up, so it fails the tight gate. Full diagnostic trail is on #2450 — including two probe results worth having regardless (the combine sums every contribution exactly, and its FP32->BF16 store does not round to nearest). Not merging it until it's green or the GB rows are explicitly scoped as unsupported coverage. 中文：你说得对，这件事已经在推进 —— #2450 新增了 它目前仍是 draft 而非就绪状态：adapter 已能在真实 GB200 硬件上运行，路由与完整性均正确，但其 combine 从 T=32 起与正确性 oracle 相差数个 ulp，无法通过严格门禁。完整的诊断记录都在 #2450 上 —— 其中两项探测结论无论如何都有价值（该 combine 会精确累加每一个贡献；其 FP32->BF16 写出并非就近舍入）。在它变绿、或 GB 行被明确标记为 unsupported coverage 之前，不会合入。 |
2 participants
Why
An audit of the SKU registry against what inference engines can actually select found that two of the four benchmarked transports are not selectable in either engine. Cross-referencing vLLM's
--all2all-backendand SGLang's--moe-a2a-backend:deepep_v2deepepmori_high_throughput/mori_low_latencymoriThat is fine in itself — both are real transports and the walls they probe are worth knowing — but nothing in the artifact or the docs said so. And it matters most where it is least visible: on B300, GB200 and GB300 the
candidateNCCL EP is the only low-latency row, so those three SKUs publish decode numbers no deployment can reproduce.What changed
Maturity is carried, not implied.
EPBackend.maturityisproductionorcandidate, each adapter declares its own, andimplementation.maturitylands in every case-attempt artifact.configs/platform_config.jsoncarries the same map for the matrix and the docs. Two copies of one fact drift silently, sotests/test_matrix.pypins coverage, a closed vocabulary, and agreement between them — reading the adapter side out of source withast, because importing an adapter pulls in torch and the vendor EP library that the test environment does not carry.The README was wrong about NCCL EP low latency, in two places. It stated the
LOW_LATENCYalgorithm "has no enabled cell" and that "NCCL EP has no low-latency row on any SKU". CollectiveX: one nccl-ep handle per group, and restore the low-latency rows #2407 enabled it on all six NVIDIA SKUs and changed onlyplatform_config.json, so the docs have contradicted the registry since it merged. Anyone reading the docs to interpret published decode numbers was told those rows do not exist. Corrected, and the B300/GB200/GB300 decode gap is now stated where a reader meets it.B300 DeepEP V2 low latency is documented as an unsupported coverage row. It had never carried one and no reason was recorded anywhere in the repo.
The B300 measurement
Run on b300-002 (single node, EP8, decode, bf16), driving the real launcher:
The legacy
deep_ep.Bufferself-enables NVSHMEM IBGDA even for a single-node EP8 run, and the transport aborts during setup. This is not the/dev/gdrdrvgap previously suspected — address-handle creation fails because a single-node run scrubs the HCA selector, so NVSHMEM auto-picks a fabric it cannot form an AH on.Since single-node EP8 needs no IB at all,
NVSHMEM_DISABLE_IB=1is the candidate fix (the adapter already passesallow_nvlink_for_low_latency_mode=True). It is untested: the two runs that would have proven it died during staging before reaching a GPU, and the partition then went fully allocated. So the row stays out until a run earns it, and the reason is now recorded rather than folklore.Verification
py_compileclean.backend_maturitymap.中文说明
为什么
对照引擎实际可选的后端排查 SKU 注册表后发现，我们基准测试的四个传输实现中有两个在任一引擎里都无法被选中。对照 vLLM 的
--all2all-backend与 SGLang 的--moe-a2a-backend：deepep_v2deepepmori_high_throughput/mori_low_latencymori这本身没有问题 —— 两者都是真实的传输实现，它们探明的边界也有价值 —— 但产物与文档都没有说明这一点。而问题最严重之处恰恰最不显眼：在 B300、GB200、GB300 上，属于 candidate 的 NCCL EP 是唯一的低延迟行，因此这三个 SKU 发布的解码数据没有任何真实部署能够复现。
改了什么
成熟度被显式携带，而非靠意会。
EPBackend.maturity取值production或candidate，由各 adapter 自行声明，并以implementation.maturity写入每一份 case-attempt 产物；configs/platform_config.json保存同一份映射供矩阵与文档使用。同一事实的两份副本会悄悄漂移，因此tests/test_matrix.py固定了覆盖面、封闭取值集合以及两者的一致性 —— adapter 一侧用ast从源码读取，因为直接 import adapter 会连带引入测试环境中并不存在的 torch 与厂商 EP 库。README 关于 NCCL EP 低延迟的描述在两处是错的。 它声称
LOW_LATENCY算法「没有启用的单元格」，以及「NCCL EP 在任何 SKU 上都没有低延迟行」。CollectiveX: one nccl-ep handle per group, and restore the low-latency rows #2407 已在全部六个 NVIDIA SKU 上启用该行，且只修改了platform_config.json，因此自其合入起文档就一直与注册表矛盾 —— 任何依据文档解读已发布解码数据的人，都会被告知这些行并不存在。现已更正，并把 B300/GB200/GB300 的解码缺口写在读者会先读到的位置。B300 的 DeepEP V2 低延迟被记录为 unsupported coverage row。 此前它从未有过该行，仓库中也没有任何地方记录原因。
B300 实测结果
在 b300-002 上（单节点、EP8、decode、bf16），通过真实 launcher 驱动：
legacy
deep_ep.Buffer即使在单节点 EP8 下也会自行启用 NVSHMEM IBGDA，该传输在初始化阶段即失败。这并不是此前怀疑的/dev/gdrdrv缺失问题 —— 真正原因是单节点运行会清除 HCA 选择器，NVSHMEM 自动挑中了一张无法建立 AH 的网卡。既然单节点 EP8 根本不需要 IB，
NVSHMEM_DISABLE_IB=1就是候选修复方案（adapter 已经传入allow_nvlink_for_low_latency_mode=True）。但该方案尚未验证：用于证明它的两次运行都在 staging 阶段退出、未触及 GPU，随后分区被完全占满。因此在有运行结果证明之前该行暂不启用，但原因已被记录下来，不再只是口口相传。验证
py_compile通过。backend_maturity映射。