From 98ef7f6a82beb4b9aa52a96e8ab20a6927446100 Mon Sep 17 00:00:00 2001 From: Oseltamivir <58582368+Oseltamivir@users.noreply.github.com> Date: Sun, 2 Aug 2026 11:25:33 +0800 Subject: [PATCH 1/3] CollectiveX: label backend maturity and correct the low-latency docs MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Three things that all turn on the same question — can anyone actually run this transport? An audit against vLLM's `--all2all-backend` and SGLang's `--moe-a2a-backend` found that two of the four benchmarked transports are not selectable in either engine. DeepEP V2 and MoRI are (`deepep_v2`/`deepep`, `mori_*`/`mori`); UCCL-EP and NCCL EP are not. That is fine for UCCL-EP and NCCL EP as candidates — they are real transports and the walls they probe are worth knowing — but nothing in the artifact or the docs said so, and on B300, GB200 and GB300 the `candidate` NCCL EP is the ONLY low-latency row, so those three SKUs publish decode numbers no deployment can reproduce. 1. **Maturity is now carried, not implied.** `EPBackend.maturity` is `production` or `candidate`, each adapter declares its own, and `implementation.maturity` lands in every case-attempt. `configs/platform_config.json` carries the same map for the matrix and the docs. Two copies of one fact drift silently, so `tests/test_matrix.py` pins coverage, a closed vocabulary, and agreement — reading the adapter side out of source with `ast`, because importing an adapter pulls in torch and the vendor EP library. 2. **The README was wrong about NCCL EP low latency.** It stated in two places that the `LOW_LATENCY` algorithm "has no enabled cell" / "no low-latency row on any SKU". #2407 enabled it on all six NVIDIA SKUs and changed only the config, so the docs have been contradicting the registry since. Corrected, and the B300/GB200/GB300 decode gap is now stated where a reader will meet it. 3. **B300 DeepEP V2 low latency is documented as an unsupported coverage row.** It had never carried one and no reason was recorded. Measured on b300-002: the legacy Buffer self-enables NVSHMEM IBGDA even for single-node EP8, and the transport aborts in setup — `ibgda.cpp:2234 NULL value Unable to create ah` -> `create DCT share err` -> `connect EPS failed` -> `nvshmem setup connections failed`, rc255 on all eight ranks. This is NOT the /dev/gdrdrv gap previously suspected; address-handle creation fails because a single-node run scrubs the HCA selector and NVSHMEM auto-picks a fabric it cannot form an AH on. Since single-node EP8 needs no IB, `NVSHMEM_DISABLE_IB=1` is the candidate fix, but the two runs that would have proven it died in staging before reaching a GPU and the partition then filled, so the row stays out until a run earns it. 中文:为后端标注成熟度(maturity),并修正低延迟相关文档。 这三件事都取决于同一个问题:这个传输实现究竟有没有人能真正跑起来? 对照 vLLM 的 `--all2all-backend` 与 SGLang 的 `--moe-a2a-backend` 做的排查发现,我们基准测试的 四个传输实现中有两个在任一引擎里都无法被选中。DeepEP V2 与 MoRI 可以(`deepep_v2`/`deepep`、 `mori_*`/`mori`);UCCL-EP 与 NCCL EP 不行。作为候选(candidate)这本身没问题 —— 它们是真实的传输 实现,其探明的边界也有价值 —— 但产物和文档都没有说明这一点;而在 B300、GB200、GB300 上,属于 candidate 的 NCCL EP 是**唯一**的低延迟行,于是这三个 SKU 发布的解码数据没有任何真实部署能复现。 1. **成熟度现在被显式携带,而不是靠读者意会。** `EPBackend.maturity` 取值 `production` 或 `candidate`,由各 adapter 自行声明,并以 `implementation.maturity` 写入每一份 case-attempt 产物;`configs/platform_config.json` 保存同一份映射供矩阵与文档使用。同一事实的两份副本会悄悄 漂移,因此 `tests/test_matrix.py` 固定了覆盖面、封闭取值集合与两者的一致性 —— adapter 一侧用 `ast` 从源码读取,因为直接 import adapter 会连带引入 torch 和厂商 EP 库。 2. **README 对 NCCL EP 低延迟的描述是错的。** 它在两处声称 `LOW_LATENCY` 算法「没有启用的单元 格」「在任何 SKU 上都没有低延迟行」。#2407 已在全部六个 NVIDIA SKU 上启用该行,且只改了配置, 于是文档自那时起一直与注册表相互矛盾。现已更正,并把 B300/GB200/GB300 的解码缺口写在读者会先 读到的位置。 3. **B300 的 DeepEP V2 低延迟被记录为 unsupported coverage row。** 此前它从未有过该行,也没有留下 原因。在 b300-002 上实测:legacy Buffer 即使在单节点 EP8 下也会自行启用 NVSHMEM IBGDA,而该传输 在初始化阶段就失败 —— `ibgda.cpp:2234 NULL value Unable to create ah` -> `create DCT share err` -> `connect EPS failed` -> `nvshmem setup connections failed`,八个 rank 全部 rc255。这**不是** 此前怀疑的 /dev/gdrdrv 缺失问题;真正原因是单节点运行会清除 HCA 选择器,NVSHMEM 自动挑到一张无法 建立 AH 的网卡。既然单节点 EP8 根本不需要 IB,`NVSHMEM_DISABLE_IB=1` 是候选修复方案;但用于验证 它的两次运行都在 staging 阶段就退出、未触及 GPU,随后分区被占满,因此在有运行结果证明之前该行 暂不启用。 --- experimental/CollectiveX/README.md | 23 ++++---- experimental/CollectiveX/bench/ep_backend.py | 8 +++ .../CollectiveX/bench/ep_deepep_v2.py | 1 + experimental/CollectiveX/bench/ep_harness.py | 4 ++ experimental/CollectiveX/bench/ep_mori.py | 1 + experimental/CollectiveX/bench/ep_nccl.py | 1 + experimental/CollectiveX/bench/ep_uccl.py | 1 + .../CollectiveX/configs/platform_config.json | 1 + experimental/CollectiveX/docs/methodology.md | 6 +- experimental/CollectiveX/sweep_matrix.py | 4 ++ experimental/CollectiveX/tests/test_matrix.py | 55 +++++++++++++++++++ 11 files changed, 94 insertions(+), 11 deletions(-) diff --git a/experimental/CollectiveX/README.md b/experimental/CollectiveX/README.md index 1e1f0ce521..ceafd7146b 100644 --- a/experimental/CollectiveX/README.md +++ b/experimental/CollectiveX/README.md @@ -28,10 +28,13 @@ run in one of two modes: pure-intranode, same compact layout and unweighted rank-sum combine as `IntraNode`). It is a decode-phase-only, per-SKU-capability-gated addition whose runnable set differs from `normal`'s, so it is enabled from each SKU's `ll_backends` registry entry (currently DeepEP V2 EP8 on H100/H200/B200, - MoRI EP8 on MI300X/MI325X/MI355X, and UCCL-EP EP8 on H100/H200/B200 only — UCCL's low-latency kernel - trips a warp-group assertion on AMD's CU count, so the AMD SKUs keep UCCL-EP normal mode without LL; - NCCL EP has no low-latency row on any SKU while its decode kernels carry - [NVIDIA/nccl#2303](https://github.com/NVIDIA/nccl/issues/2303)). + MoRI EP8 on MI300X/MI325X/MI355X, UCCL-EP EP8 on H100/H200/B200 only — UCCL's low-latency kernel + trips a warp-group assertion on AMD's CU count, so the AMD SKUs keep UCCL-EP normal mode without LL — + and NCCL EP EP8 on all six NVIDIA SKUs, restored once the single-handle fix removed the + [NVIDIA/nccl#2303](https://github.com/NVIDIA/nccl/issues/2303) signal aliasing that had wedged them. + B300, GB200 and GB300 carry NCCL EP as their *only* low-latency row, and NCCL EP is a `candidate` + transport (no engine exposes a selector for it), so those three SKUs have no production decode + coverage; B300's DeepEP V2 low-latency row is an unsupported coverage row, see the backend table). Scoped single-node EP8 runs over the intra-node NVLink/XGMI low-latency path (no `/dev/gdrdrv` needed — validated on H200 with it absent); NVSHMEM/IBGDA on the wire is only a multi-node scale-out (EP16) concern. @@ -67,12 +70,12 @@ frozen digest or locked case count. Physical host count does not determine scope: both GB topologies stay inside one 72-GPU MNNVL scale-up domain. -| Backend | Current scope | -|---|---| -| DeepEP V2 | `normal` mode is PR #605 `ElasticBuffer` plus exact upstream #630 and #640 fixes: LSA for scale-up and GIN for x86 EP16 scale-out. FP8 dispatch via `use_fp8_dispatch` (blockwise e4m3fn) alongside BF16. `low-latency` mode is the legacy `deep_ep.Buffer` IBGDA decode kernels (per-expert padded layout, weighted combine, `use_fp8` e4m3fn), decode/EP8 only | -| MoRI | `normal` mode uses the direct `IntraNode` kernel for scale-up EP8 on every CDNA SKU and pins `InterNodeV1` for EP16 over 2x8 XGMI + RDMA. `low-latency` mode selects the `IntraNodeLL` decode kernel (single-call, pure-intranode, same compact layout and unweighted combine as `IntraNode`), decode/EP8 only. FP8 dispatch is caller-prequantized (per-SKU e4m3fnuz on gfx942, e4m3fn on gfx950); combine stays BF16 (`quant_type=none`) alongside BF16 dispatch | -| UCCL-EP | [UCCL](https://github.com/uccl-project/uccl) EP: a drop-in, API-identical DeepEP replacement whose CPU proxies issue GPUDirect RDMA over plain `libibverbs` (no NVSHMEM/IBGDA), with software message ordering, atomics, and flow control; scale-up is single-node `cudaIpc` over NVLink/XGMI (never MNNVL). `normal` mode is the legacy `Buffer` `dispatch`/`combine` (unweighted rank-sum); `low-latency` reuses the legacy `low_latency_dispatch`/`low_latency_combine` decode kernels (weighted combine), decode/EP8 only. FP8 dispatch is caller-prequantized in `normal` mode (blockwise e4m3fn, per-SKU e4m3fnuz on gfx942); in `low-latency` mode the caller sends BF16 and the decode kernel quantizes to e4m3 internally (`use_fp8`). Combine is BF16. Runs on NVIDIA and AMD (H100/H200/B200 + MI300X/MI325X/MI355X), EP8 scale-up. Cross-node EP16 is functional (the internode RDMA path connects and the light case passes correctness) but its CPU-proxy throughput overruns the standardized per-case wall-clock budget on heavy token counts, so EP16 is an unsupported coverage row for now | -| NCCL EP | [NCCL EP](https://github.com/NVIDIA/nccl/tree/master/contrib/nccl_ep): NVIDIA's native MoE dispatch/combine on the NCCL Device API — LSA (NVLink load/store) intra-node, GIN (GPU-Initiated Networking) inter-node — driven through the `nccl4py` bindings. `normal` mode selects the `HIGH_THROUGHPUT` algorithm (FLAT `[N, hidden]` receive, unweighted rank-sum combine); the `LOW_LATENCY` algorithm is implemented in the adapter but has no enabled cell (see the `ll_backends` note above). BF16 only: NCCL EP's FP8 machinery exists upstream but its RELEASE.md lists it unsupported/untested, so no FP8 case is emitted. NVIDIA-only and CUDA 13 only. EP8 scale-up on H100/H200/B200/B300 plus EP8 and EP16 on GB200/GB300, where EP16 stays inside the MNNVL scale-up domain. x86 EP16 scale-out is an unsupported coverage row: the cross-node GIN path faults inside `nccl_ep.cc` identically on RoCE and IB across four SKUs, a GDAKI limit rather than a fabric-selection one | +| Backend | Engine availability | Current scope | +|---|---|---| +| DeepEP V2 | `production` — vLLM `--all2all-backend deepep_v2`, SGLang `--moe-a2a-backend deepep` | `normal` mode is PR #605 `ElasticBuffer` plus exact upstream #630 and #640 fixes: LSA for scale-up and GIN for x86 EP16 scale-out. FP8 dispatch via `use_fp8_dispatch` (blockwise e4m3fn) alongside BF16. `low-latency` mode is the legacy `deep_ep.Buffer` IBGDA decode kernels (per-expert padded layout, weighted combine, `use_fp8` e4m3fn), decode/EP8 only. B300 is an unsupported coverage row in `low-latency`: the legacy Buffer self-enables NVSHMEM IBGDA even for a single-node EP8 run, and on B300 the transport aborts during setup — `ibgda.cpp:2234 NULL value Unable to create ah` -> `create DCT share err` -> `connect EPS failed` -> `nvshmem setup connections failed`, rc255 on all eight ranks (measured 2026-08-02 on b300-002). Single-node EP8 needs no IB at all, so forcing the NVLink low-latency path (`NVSHMEM_DISABLE_IB=1`, the adapter already passes `allow_nvlink_for_low_latency_mode=True`) is the candidate fix; it is untested, so the row stays out until a run proves it | +| MoRI | `production` — vLLM `--all2all-backend mori_*`, SGLang `--moe-a2a-backend mori` | `normal` mode uses the direct `IntraNode` kernel for scale-up EP8 on every CDNA SKU and pins `InterNodeV1` for EP16 over 2x8 XGMI + RDMA. `low-latency` mode selects the `IntraNodeLL` decode kernel (single-call, pure-intranode, same compact layout and unweighted combine as `IntraNode`), decode/EP8 only. FP8 dispatch is caller-prequantized (per-SKU e4m3fnuz on gfx942, e4m3fn on gfx950); combine stays BF16 (`quant_type=none`) alongside BF16 dispatch | +| UCCL-EP | `candidate` — no engine exposes a UCCL-EP selector | [UCCL](https://github.com/uccl-project/uccl) EP: a drop-in, API-identical DeepEP replacement whose CPU proxies issue GPUDirect RDMA over plain `libibverbs` (no NVSHMEM/IBGDA), with software message ordering, atomics, and flow control; scale-up is single-node `cudaIpc` over NVLink/XGMI (never MNNVL). `normal` mode is the legacy `Buffer` `dispatch`/`combine` (unweighted rank-sum); `low-latency` reuses the legacy `low_latency_dispatch`/`low_latency_combine` decode kernels (weighted combine), decode/EP8 only. FP8 dispatch is caller-prequantized in `normal` mode (blockwise e4m3fn, per-SKU e4m3fnuz on gfx942); in `low-latency` mode the caller sends BF16 and the decode kernel quantizes to e4m3 internally (`use_fp8`). Combine is BF16. Runs on NVIDIA and AMD (H100/H200/B200 + MI300X/MI325X/MI355X), EP8 scale-up. Cross-node EP16 is functional (the internode RDMA path connects and the light case passes correctness) but its CPU-proxy throughput overruns the standardized per-case wall-clock budget on heavy token counts, so EP16 is an unsupported coverage row for now | +| NCCL EP | `candidate` — NVIDIA's own library, but no engine exposes an NCCL-EP selector | [NCCL EP](https://github.com/NVIDIA/nccl/tree/master/contrib/nccl_ep): NVIDIA's native MoE dispatch/combine on the NCCL Device API — LSA (NVLink load/store) intra-node, GIN (GPU-Initiated Networking) inter-node — driven through the `nccl4py` bindings. `normal` mode selects the `HIGH_THROUGHPUT` algorithm (FLAT `[N, hidden]` receive, unweighted rank-sum combine); the `LOW_LATENCY` algorithm carries an EP8 `ll_backends` row on all six NVIDIA SKUs, restored once the single-handle fix removed the NVIDIA/nccl#2303 signal aliasing. BF16 only: NCCL EP's FP8 machinery exists upstream but its RELEASE.md lists it unsupported/untested, so no FP8 case is emitted. NVIDIA-only and CUDA 13 only. EP8 scale-up on H100/H200/B200/B300 plus EP8 and EP16 on GB200/GB300, where EP16 stays inside the MNNVL scale-up domain. x86 EP16 scale-out is an unsupported coverage row: the cross-node GIN path faults inside `nccl_ep.cc` identically on RoCE and IB across four SKUs, a GDAKI limit rather than a fabric-selection one | DeepEP V2 means the `ElasticBuffer` implementation introduced by [DeepEP PR #605](https://github.com/deepseek-ai/DeepEP/pull/605), not a newer legacy `Buffer` build. diff --git a/experimental/CollectiveX/bench/ep_backend.py b/experimental/CollectiveX/bench/ep_backend.py index faf2726e15..0c791cc526 100644 --- a/experimental/CollectiveX/bench/ep_backend.py +++ b/experimental/CollectiveX/bench/ep_backend.py @@ -67,6 +67,14 @@ class EPBackend(abc.ABC): """ name: str = "" + # Whether a production inference engine can actually select this transport today: + # "production" = exposed as an all-to-all backend by vLLM (`--all2all-backend`) or + # SGLang (`--moe-a2a-backend`); "candidate" = a real transport we benchmark, but one + # no engine ships a selector for, so its numbers describe the library rather than a + # deployable configuration. Emitted per case-attempt so a reader can tell the two + # apart; `configs/platform_config.json` carries the same map for the matrix and the + # docs, and tests/test_matrix.py holds the two in agreement. + maturity: str = "" SUPPORTED_MODES: tuple = ("normal",) # Dispatch precisions the adapter realizes. BF16 is the universal control; an # adapter that also sends an FP8-quantized dispatch payload widens this. diff --git a/experimental/CollectiveX/bench/ep_deepep_v2.py b/experimental/CollectiveX/bench/ep_deepep_v2.py index 8708d9886d..f74cfcf873 100644 --- a/experimental/CollectiveX/bench/ep_deepep_v2.py +++ b/experimental/CollectiveX/bench/ep_deepep_v2.py @@ -116,6 +116,7 @@ def _require_runtime() -> None: class DeepEPV2Backend(EPBackend): name = "deepep-v2" + maturity = "production" # vLLM --all2all-backend deepep_v2; SGLang --moe-a2a-backend deepep # Two kernel families under one adapter, selected by mode: # normal -> PR #605 ElasticBuffer (LSA vs hybrid GIN are transport paths, not # kernel families); rank-deduplicated unweighted-rank-sum combine. diff --git a/experimental/CollectiveX/bench/ep_harness.py b/experimental/CollectiveX/bench/ep_harness.py index 387d4a44a4..71b58031fe 100644 --- a/experimental/CollectiveX/bench/ep_harness.py +++ b/experimental/CollectiveX/bench/ep_harness.py @@ -1131,6 +1131,10 @@ def run_sweep(args, backend, torch, dist, device, rank: int, world_size: int) -> # EPBackend.fp8_consume. Only meaningful when the case dispatches FP8. "fp8_consume": getattr(backend, "fp8_consume", None), "kernel_generation": kernel_generation(backend), + # Whether an inference engine can select this transport today — see + # EPBackend.maturity. A "candidate" row measures the library, not a + # deployable configuration, and must not be read as one. + "maturity": getattr(backend, "maturity", None) or "unknown", "name": backend.name, }, "topology": { diff --git a/experimental/CollectiveX/bench/ep_mori.py b/experimental/CollectiveX/bench/ep_mori.py index e3adeeb71b..2cda3f974c 100644 --- a/experimental/CollectiveX/bench/ep_mori.py +++ b/experimental/CollectiveX/bench/ep_mori.py @@ -38,6 +38,7 @@ def _project_local_metadata(torch_module, raw_expert_ids, raw_weights, rank, exp class MoRIBackend(EPBackend): name = "mori" + maturity = "production" # vLLM --all2all-backend mori_*; SGLang --moe-a2a-backend mori SUPPORTED_MODES = ("normal", "low-latency") SUPPORTED_PRECISIONS = ("bf16", "fp8") combine_needs_redispatch = True diff --git a/experimental/CollectiveX/bench/ep_nccl.py b/experimental/CollectiveX/bench/ep_nccl.py index 14b4c2de9e..3b1010fe7b 100644 --- a/experimental/CollectiveX/bench/ep_nccl.py +++ b/experimental/CollectiveX/bench/ep_nccl.py @@ -65,6 +65,7 @@ class NCCLEPBackend(EPBackend): name = "nccl-ep" + maturity = "candidate" # NVIDIA's library, but no engine exposes an NCCL-EP selector # One library, two algorithms selected by args.mode. kernel_generation and the combine # semantics are switched to their LL values in __init__ (mirrors ep_deepep_v2). # normal -> HT / FLAT layout / unweighted-rank-sum combine. diff --git a/experimental/CollectiveX/bench/ep_uccl.py b/experimental/CollectiveX/bench/ep_uccl.py index 821b23c25d..9ef1fa7899 100644 --- a/experimental/CollectiveX/bench/ep_uccl.py +++ b/experimental/CollectiveX/bench/ep_uccl.py @@ -123,6 +123,7 @@ def _normal_num_sms() -> int: class UCCLEPBackend(EPBackend): name = "uccl-ep" + maturity = "candidate" # no engine exposes a UCCL-EP all-to-all selector # One legacy Buffer under two modes, selected by args.mode: # normal -> get_dispatch_layout/dispatch/combine; unweighted rank-sum combine. # low-latency -> low_latency_dispatch/combine decode kernels; source-side weighted combine. diff --git a/experimental/CollectiveX/configs/platform_config.json b/experimental/CollectiveX/configs/platform_config.json index 3babb66536..018c79805a 100644 --- a/experimental/CollectiveX/configs/platform_config.json +++ b/experimental/CollectiveX/configs/platform_config.json @@ -1,4 +1,5 @@ { + "backend_maturity": {"deepep-v2": "production", "mori": "production", "uccl-ep": "candidate", "nccl-ep": "candidate"}, "platforms": { "h100-dgxc": { "arch": "sm90", diff --git a/experimental/CollectiveX/docs/methodology.md b/experimental/CollectiveX/docs/methodology.md index 4ec82c7258..2a8840e8b0 100644 --- a/experimental/CollectiveX/docs/methodology.md +++ b/experimental/CollectiveX/docs/methodology.md @@ -202,7 +202,11 @@ One raw case document carries `record_type: "case-attempt"` and the single `vers - `workload`: `cross_rank_consistent`, whether the routing trace was proven identical across ranks; - `measurement`: dispatch/combine dtype (the realized wire formats — combine always BF16, dispatch BF16 or the SKU's FP8 format) and semantics, `sampling`, and the per-point `rows`; -- `implementation`: backend name and kernel generation; +- `implementation`: backend name, kernel generation, and `maturity` — whether a production + inference engine can select this transport today (`production` = exposed by vLLM's + `--all2all-backend` or SGLang's `--moe-a2a-backend`; `candidate` = a real transport we + benchmark that no engine ships a selector for, so its numbers describe the library rather + than a deployable configuration). The same map is in the registry's `backend_maturity`; - `topology`: requested SKU/product, placement, nodes, scale-up domain, transport, and world size; - `provenance`: the mounted image tag and source SHA; and - `outcome`: `status` (`success` or `invalid`) and `reasons`. diff --git a/experimental/CollectiveX/sweep_matrix.py b/experimental/CollectiveX/sweep_matrix.py index 9a38f7f9ee..9a3dbc0c11 100644 --- a/experimental/CollectiveX/sweep_matrix.py +++ b/experimental/CollectiveX/sweep_matrix.py @@ -26,6 +26,10 @@ def _load_config(name: str) -> dict[str, Any]: SWEEP = _load_config("sweep.json") PLATFORMS = _load_config("platform_config.json")["platforms"] +# Whether an inference engine can select each transport today (see EPBackend.maturity). +# Consumed by the docs and the matrix; each adapter carries the same value for the +# artifact it writes, and tests/test_matrix.py holds the two in agreement. +BACKEND_MATURITY = _load_config("platform_config.json")["backend_maturity"] SWEEP_BACKENDS = tuple(dict.fromkeys( backend for platform in PLATFORMS.values() for backend in platform["backends"] )) diff --git a/experimental/CollectiveX/tests/test_matrix.py b/experimental/CollectiveX/tests/test_matrix.py index 83d3c2a4b7..b799506b17 100644 --- a/experimental/CollectiveX/tests/test_matrix.py +++ b/experimental/CollectiveX/tests/test_matrix.py @@ -281,5 +281,60 @@ def test_invalid_filters_fail_closed(self): sweep_matrix.resolve_matrix(**options) +class BackendMaturityTests(unittest.TestCase): + """The registry map and each adapter's `maturity` are two copies of one fact. + + The registry drives the matrix and the docs; the adapter attribute is what lands in + every case-attempt artifact. They are read by different consumers and can drift + silently, so pin both: complete coverage, a closed vocabulary, and agreement. The + adapter side is read from source rather than imported, because importing an adapter + pulls in torch and the vendor EP library, which the test image does not carry. + """ + + VOCABULARY = {"production", "candidate"} + + @staticmethod + def _declared_in_source(): + """{backend name: maturity} parsed from the adapter class bodies.""" + import ast + + declared = {} + for path in sorted((ROOT / "bench").glob("ep_*.py")): + tree = ast.parse(path.read_text()) + for node in ast.walk(tree): + if not isinstance(node, ast.ClassDef): + continue + literals = {} + for statement in node.body: + if not isinstance(statement, ast.Assign): + continue + if not isinstance(statement.value, ast.Constant): + continue + for target in statement.targets: + if isinstance(target, ast.Name): + literals[target.id] = statement.value.value + # The abstract base declares both as empty defaults; skip it. + if literals.get("name") and "maturity" in literals: + declared[literals["name"]] = literals["maturity"] + return declared + + def test_registry_covers_every_dispatched_backend(self): + maturity = sweep_matrix.BACKEND_MATURITY + for sku, platform in sweep_matrix.PLATFORMS.items(): + for backend in platform["backends"]: + with self.subTest(sku=sku, backend=backend): + self.assertIn(backend, maturity) + self.assertIn(maturity[backend], self.VOCABULARY) + + def test_adapters_and_registry_agree(self): + declared = self._declared_in_source() + # Every backend the matrix can dispatch must declare a maturity in its adapter, + # or the artifact it writes would say "unknown" while the registry says otherwise. + for backend, expected in sweep_matrix.BACKEND_MATURITY.items(): + with self.subTest(backend=backend): + self.assertIn(backend, declared) + self.assertEqual(declared[backend], expected) + + if __name__ == "__main__": unittest.main() From 6713de09f7014a16f5dde67bd4aab5e029d0e02c Mon Sep 17 00:00:00 2001 From: Oseltamivir <58582368+Oseltamivir@users.noreply.github.com> Date: Sun, 2 Aug 2026 13:54:09 +0800 Subject: [PATCH 2/3] CollectiveX: record that NVSHMEM_DISABLE_IB does not fix B300 low latency MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The previous commit called `NVSHMEM_DISABLE_IB=1` the candidate fix for B300's DeepEP V2 low-latency row and said it was untested. It has now been tested on b300-011 and it does not work: the run fails identically with the variable set and unset — 16 `Unable to create ah` failures either way. The legacy Buffer self-enables IBGDA regardless, so the environment variable never gets a say. That turns a guess into a measured dead end, which is worth writing down: the next attempt has to stop the low-latency path from initializing the NVSHMEM RDMA transport at all, or point it at a fabric on which B300 can form an address handle — not just ask NVSHMEM nicely to skip IB. 中文:记录 `NVSHMEM_DISABLE_IB` 无法修复 B300 低延迟问题。 上一个提交把 `NVSHMEM_DISABLE_IB=1` 称为 B300 DeepEP V2 低延迟行的候选修复方案,并注明尚未验证。 现已在 b300-011 上完成验证:该方案无效 —— 无论设置与否,运行都以完全相同的方式失败,两种情况下 均为 16 次 `Unable to create ah`。legacy Buffer 会自行启用 IBGDA,因此这个环境变量根本不起作用。 这把一个猜测变成了有实测支撑的死路,值得记录下来:下一次尝试必须让低延迟路径彻底不初始化 NVSHMEM RDMA 传输,或将其指向 B300 能够建立 address handle 的 fabric —— 而不是只是「请求」 NVSHMEM 跳过 IB。 --- experimental/CollectiveX/README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/experimental/CollectiveX/README.md b/experimental/CollectiveX/README.md index ceafd7146b..febb08d9ad 100644 --- a/experimental/CollectiveX/README.md +++ b/experimental/CollectiveX/README.md @@ -72,7 +72,7 @@ scale-up domain. | Backend | Engine availability | Current scope | |---|---|---| -| DeepEP V2 | `production` — vLLM `--all2all-backend deepep_v2`, SGLang `--moe-a2a-backend deepep` | `normal` mode is PR #605 `ElasticBuffer` plus exact upstream #630 and #640 fixes: LSA for scale-up and GIN for x86 EP16 scale-out. FP8 dispatch via `use_fp8_dispatch` (blockwise e4m3fn) alongside BF16. `low-latency` mode is the legacy `deep_ep.Buffer` IBGDA decode kernels (per-expert padded layout, weighted combine, `use_fp8` e4m3fn), decode/EP8 only. B300 is an unsupported coverage row in `low-latency`: the legacy Buffer self-enables NVSHMEM IBGDA even for a single-node EP8 run, and on B300 the transport aborts during setup — `ibgda.cpp:2234 NULL value Unable to create ah` -> `create DCT share err` -> `connect EPS failed` -> `nvshmem setup connections failed`, rc255 on all eight ranks (measured 2026-08-02 on b300-002). Single-node EP8 needs no IB at all, so forcing the NVLink low-latency path (`NVSHMEM_DISABLE_IB=1`, the adapter already passes `allow_nvlink_for_low_latency_mode=True`) is the candidate fix; it is untested, so the row stays out until a run proves it | +| DeepEP V2 | `production` — vLLM `--all2all-backend deepep_v2`, SGLang `--moe-a2a-backend deepep` | `normal` mode is PR #605 `ElasticBuffer` plus exact upstream #630 and #640 fixes: LSA for scale-up and GIN for x86 EP16 scale-out. FP8 dispatch via `use_fp8_dispatch` (blockwise e4m3fn) alongside BF16. `low-latency` mode is the legacy `deep_ep.Buffer` IBGDA decode kernels (per-expert padded layout, weighted combine, `use_fp8` e4m3fn), decode/EP8 only. B300 is an unsupported coverage row in `low-latency`: the legacy Buffer self-enables NVSHMEM IBGDA even for a single-node EP8 run, and on B300 the transport aborts during setup — `ibgda.cpp:2234 NULL value Unable to create ah` -> `create DCT share err` -> `connect EPS failed` -> `nvshmem setup connections failed`, rc255 on all eight ranks (measured 2026-08-02 on b300-002). Single-node EP8 needs no IB at all, so the obvious fix was to force the NVLink low-latency path with `NVSHMEM_DISABLE_IB=1` (the adapter already passes `allow_nvlink_for_low_latency_mode=True`). That was tested on b300-011 and **does not work**: the run fails identically, 16 address-handle creation failures with the variable set and with it unset. The legacy Buffer self-enables IBGDA regardless, so the env var never gets a say. A fix has to stop the LL path from initializing the NVSHMEM RDMA transport at all, or point it at a fabric on which B300 can form an AH | | MoRI | `production` — vLLM `--all2all-backend mori_*`, SGLang `--moe-a2a-backend mori` | `normal` mode uses the direct `IntraNode` kernel for scale-up EP8 on every CDNA SKU and pins `InterNodeV1` for EP16 over 2x8 XGMI + RDMA. `low-latency` mode selects the `IntraNodeLL` decode kernel (single-call, pure-intranode, same compact layout and unweighted combine as `IntraNode`), decode/EP8 only. FP8 dispatch is caller-prequantized (per-SKU e4m3fnuz on gfx942, e4m3fn on gfx950); combine stays BF16 (`quant_type=none`) alongside BF16 dispatch | | UCCL-EP | `candidate` — no engine exposes a UCCL-EP selector | [UCCL](https://github.com/uccl-project/uccl) EP: a drop-in, API-identical DeepEP replacement whose CPU proxies issue GPUDirect RDMA over plain `libibverbs` (no NVSHMEM/IBGDA), with software message ordering, atomics, and flow control; scale-up is single-node `cudaIpc` over NVLink/XGMI (never MNNVL). `normal` mode is the legacy `Buffer` `dispatch`/`combine` (unweighted rank-sum); `low-latency` reuses the legacy `low_latency_dispatch`/`low_latency_combine` decode kernels (weighted combine), decode/EP8 only. FP8 dispatch is caller-prequantized in `normal` mode (blockwise e4m3fn, per-SKU e4m3fnuz on gfx942); in `low-latency` mode the caller sends BF16 and the decode kernel quantizes to e4m3 internally (`use_fp8`). Combine is BF16. Runs on NVIDIA and AMD (H100/H200/B200 + MI300X/MI325X/MI355X), EP8 scale-up. Cross-node EP16 is functional (the internode RDMA path connects and the light case passes correctness) but its CPU-proxy throughput overruns the standardized per-case wall-clock budget on heavy token counts, so EP16 is an unsupported coverage row for now | | NCCL EP | `candidate` — NVIDIA's own library, but no engine exposes an NCCL-EP selector | [NCCL EP](https://github.com/NVIDIA/nccl/tree/master/contrib/nccl_ep): NVIDIA's native MoE dispatch/combine on the NCCL Device API — LSA (NVLink load/store) intra-node, GIN (GPU-Initiated Networking) inter-node — driven through the `nccl4py` bindings. `normal` mode selects the `HIGH_THROUGHPUT` algorithm (FLAT `[N, hidden]` receive, unweighted rank-sum combine); the `LOW_LATENCY` algorithm carries an EP8 `ll_backends` row on all six NVIDIA SKUs, restored once the single-handle fix removed the NVIDIA/nccl#2303 signal aliasing. BF16 only: NCCL EP's FP8 machinery exists upstream but its RELEASE.md lists it unsupported/untested, so no FP8 case is emitted. NVIDIA-only and CUDA 13 only. EP8 scale-up on H100/H200/B200/B300 plus EP8 and EP16 on GB200/GB300, where EP16 stays inside the MNNVL scale-up domain. x86 EP16 scale-out is an unsupported coverage row: the cross-node GIN path faults inside `nccl_ep.cc` identically on RoCE and IB across four SKUs, a GDAKI limit rather than a fabric-selection one | From ce469ba6988e5839079f4ab7c9a39a6f073c6e97 Mon Sep 17 00:00:00 2001 From: Oseltamivir <58582368+Oseltamivir@users.noreply.github.com> Date: Sun, 2 Aug 2026 16:10:41 +0800 Subject: [PATCH 3/3] CollectiveX: fix the stale methodology Matrix note and trim the labelling diff MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Review caught the same false claim I corrected in the README surviving in docs/methodology.md's Matrix section: NCCL EP "carries no ll_backends row on any SKU ... the cells stay out of the matrix until a fixed wheel ships". #2407 enabled those rows on all six NVIDIA SKUs. Corrected, with the B300/GB200/GB300 decode gap stated there too. Also tightened the diff after re-reading every added line: * The production/candidate definition was written out three times (ep_backend, ep_harness, sweep_matrix). It now lives once on EPBackend.maturity; the other two point at it. * The B300 low-latency explanation in the backend table ran to 945 characters inside one cell. Kept the diagnostic facts — IBGDA self-enabled on a single-node run, the ibgda.cpp:2234 address-handle failure, rc255, and that NVSHMEM_DISABLE_IB=1 was tested and does not help — dropped the four-step error chain, the narrative around the attempted fix, and a speculative sentence about what a real fix would need. * Dropped a parenthetical in the ll_backends note that repeated the backend table's own candidate cell. 90 -> 89 added lines, and the ones that remain each say something the others do not. 中文:修正 methodology 中过时的 Matrix 说明,并精简标注改动的 diff。 Review 发现我已在 README 中更正的同一处错误描述仍存在于 docs/methodology.md 的 Matrix 章节: 称 NCCL EP「在任何 SKU 上都没有 ll_backends 行……相关单元格在修复版 wheel 发布前不进入矩阵」。 而 #2407 已在全部六个 NVIDIA SKU 上启用这些行。现已更正,并同时写明 B300/GB200/GB300 的解码缺口。 在逐行复查新增内容后也做了精简: * production/candidate 的定义此前被写了三遍(ep_backend、ep_harness、sweep_matrix)。现在只 保留在 EPBackend.maturity 一处,另外两处指向它。 * 后端表中 B300 低延迟的说明在单个单元格里达到 945 字符。保留了诊断事实 —— 单节点运行下 IBGDA 自行启用、ibgda.cpp:2234 的 address handle 失败、rc255,以及 NVSHMEM_DISABLE_IB=1 经测试无效 —— 删去了四步错误链、围绕尝试修复的叙述,以及关于「真正的修复需要什么」的推测性句子。 * 删去 ll_backends 说明中一处与后端表 candidate 单元格重复的插入语。 新增行数从 90 降到 89,且保留下来的每一行都在陈述其他行没有说的内容。 --- experimental/CollectiveX/README.md | 7 +++---- experimental/CollectiveX/bench/ep_backend.py | 11 ++++------- experimental/CollectiveX/bench/ep_harness.py | 4 +--- experimental/CollectiveX/docs/methodology.md | 11 ++++++----- experimental/CollectiveX/sweep_matrix.py | 4 +--- experimental/CollectiveX/tests/test_matrix.py | 11 ++++------- 6 files changed, 19 insertions(+), 29 deletions(-) diff --git a/experimental/CollectiveX/README.md b/experimental/CollectiveX/README.md index febb08d9ad..956e741d98 100644 --- a/experimental/CollectiveX/README.md +++ b/experimental/CollectiveX/README.md @@ -32,9 +32,8 @@ run in one of two modes: trips a warp-group assertion on AMD's CU count, so the AMD SKUs keep UCCL-EP normal mode without LL — and NCCL EP EP8 on all six NVIDIA SKUs, restored once the single-handle fix removed the [NVIDIA/nccl#2303](https://github.com/NVIDIA/nccl/issues/2303) signal aliasing that had wedged them. - B300, GB200 and GB300 carry NCCL EP as their *only* low-latency row, and NCCL EP is a `candidate` - transport (no engine exposes a selector for it), so those three SKUs have no production decode - coverage; B300's DeepEP V2 low-latency row is an unsupported coverage row, see the backend table). + B300, GB200 and GB300 carry the `candidate` NCCL EP as their *only* low-latency row, so those + three SKUs have no production decode coverage; B300's DeepEP V2 LL row is unsupported, see below). Scoped single-node EP8 runs over the intra-node NVLink/XGMI low-latency path (no `/dev/gdrdrv` needed — validated on H200 with it absent); NVSHMEM/IBGDA on the wire is only a multi-node scale-out (EP16) concern. @@ -72,7 +71,7 @@ scale-up domain. | Backend | Engine availability | Current scope | |---|---|---| -| DeepEP V2 | `production` — vLLM `--all2all-backend deepep_v2`, SGLang `--moe-a2a-backend deepep` | `normal` mode is PR #605 `ElasticBuffer` plus exact upstream #630 and #640 fixes: LSA for scale-up and GIN for x86 EP16 scale-out. FP8 dispatch via `use_fp8_dispatch` (blockwise e4m3fn) alongside BF16. `low-latency` mode is the legacy `deep_ep.Buffer` IBGDA decode kernels (per-expert padded layout, weighted combine, `use_fp8` e4m3fn), decode/EP8 only. B300 is an unsupported coverage row in `low-latency`: the legacy Buffer self-enables NVSHMEM IBGDA even for a single-node EP8 run, and on B300 the transport aborts during setup — `ibgda.cpp:2234 NULL value Unable to create ah` -> `create DCT share err` -> `connect EPS failed` -> `nvshmem setup connections failed`, rc255 on all eight ranks (measured 2026-08-02 on b300-002). Single-node EP8 needs no IB at all, so the obvious fix was to force the NVLink low-latency path with `NVSHMEM_DISABLE_IB=1` (the adapter already passes `allow_nvlink_for_low_latency_mode=True`). That was tested on b300-011 and **does not work**: the run fails identically, 16 address-handle creation failures with the variable set and with it unset. The legacy Buffer self-enables IBGDA regardless, so the env var never gets a say. A fix has to stop the LL path from initializing the NVSHMEM RDMA transport at all, or point it at a fabric on which B300 can form an AH | +| DeepEP V2 | `production` — vLLM `--all2all-backend deepep_v2`, SGLang `--moe-a2a-backend deepep` | `normal` mode is PR #605 `ElasticBuffer` plus exact upstream #630 and #640 fixes: LSA for scale-up and GIN for x86 EP16 scale-out. FP8 dispatch via `use_fp8_dispatch` (blockwise e4m3fn) alongside BF16. `low-latency` mode is the legacy `deep_ep.Buffer` IBGDA decode kernels (per-expert padded layout, weighted combine, `use_fp8` e4m3fn), decode/EP8 only. B300 is an unsupported coverage row in `low-latency`: the legacy Buffer self-enables NVSHMEM IBGDA even for a single-node EP8 run, and on B300 address-handle creation fails (`ibgda.cpp:2234 Unable to create ah`), rc255 on all eight ranks. `NVSHMEM_DISABLE_IB=1` does not help — the Buffer re-enables IBGDA regardless, and the run fails identically with it set and unset (measured on b300-002 and b300-011) | | MoRI | `production` — vLLM `--all2all-backend mori_*`, SGLang `--moe-a2a-backend mori` | `normal` mode uses the direct `IntraNode` kernel for scale-up EP8 on every CDNA SKU and pins `InterNodeV1` for EP16 over 2x8 XGMI + RDMA. `low-latency` mode selects the `IntraNodeLL` decode kernel (single-call, pure-intranode, same compact layout and unweighted combine as `IntraNode`), decode/EP8 only. FP8 dispatch is caller-prequantized (per-SKU e4m3fnuz on gfx942, e4m3fn on gfx950); combine stays BF16 (`quant_type=none`) alongside BF16 dispatch | | UCCL-EP | `candidate` — no engine exposes a UCCL-EP selector | [UCCL](https://github.com/uccl-project/uccl) EP: a drop-in, API-identical DeepEP replacement whose CPU proxies issue GPUDirect RDMA over plain `libibverbs` (no NVSHMEM/IBGDA), with software message ordering, atomics, and flow control; scale-up is single-node `cudaIpc` over NVLink/XGMI (never MNNVL). `normal` mode is the legacy `Buffer` `dispatch`/`combine` (unweighted rank-sum); `low-latency` reuses the legacy `low_latency_dispatch`/`low_latency_combine` decode kernels (weighted combine), decode/EP8 only. FP8 dispatch is caller-prequantized in `normal` mode (blockwise e4m3fn, per-SKU e4m3fnuz on gfx942); in `low-latency` mode the caller sends BF16 and the decode kernel quantizes to e4m3 internally (`use_fp8`). Combine is BF16. Runs on NVIDIA and AMD (H100/H200/B200 + MI300X/MI325X/MI355X), EP8 scale-up. Cross-node EP16 is functional (the internode RDMA path connects and the light case passes correctness) but its CPU-proxy throughput overruns the standardized per-case wall-clock budget on heavy token counts, so EP16 is an unsupported coverage row for now | | NCCL EP | `candidate` — NVIDIA's own library, but no engine exposes an NCCL-EP selector | [NCCL EP](https://github.com/NVIDIA/nccl/tree/master/contrib/nccl_ep): NVIDIA's native MoE dispatch/combine on the NCCL Device API — LSA (NVLink load/store) intra-node, GIN (GPU-Initiated Networking) inter-node — driven through the `nccl4py` bindings. `normal` mode selects the `HIGH_THROUGHPUT` algorithm (FLAT `[N, hidden]` receive, unweighted rank-sum combine); the `LOW_LATENCY` algorithm carries an EP8 `ll_backends` row on all six NVIDIA SKUs, restored once the single-handle fix removed the NVIDIA/nccl#2303 signal aliasing. BF16 only: NCCL EP's FP8 machinery exists upstream but its RELEASE.md lists it unsupported/untested, so no FP8 case is emitted. NVIDIA-only and CUDA 13 only. EP8 scale-up on H100/H200/B200/B300 plus EP8 and EP16 on GB200/GB300, where EP16 stays inside the MNNVL scale-up domain. x86 EP16 scale-out is an unsupported coverage row: the cross-node GIN path faults inside `nccl_ep.cc` identically on RoCE and IB across four SKUs, a GDAKI limit rather than a fabric-selection one | diff --git a/experimental/CollectiveX/bench/ep_backend.py b/experimental/CollectiveX/bench/ep_backend.py index 0c791cc526..a16d1a015b 100644 --- a/experimental/CollectiveX/bench/ep_backend.py +++ b/experimental/CollectiveX/bench/ep_backend.py @@ -67,13 +67,10 @@ class EPBackend(abc.ABC): """ name: str = "" - # Whether a production inference engine can actually select this transport today: - # "production" = exposed as an all-to-all backend by vLLM (`--all2all-backend`) or - # SGLang (`--moe-a2a-backend`); "candidate" = a real transport we benchmark, but one - # no engine ships a selector for, so its numbers describe the library rather than a - # deployable configuration. Emitted per case-attempt so a reader can tell the two - # apart; `configs/platform_config.json` carries the same map for the matrix and the - # docs, and tests/test_matrix.py holds the two in agreement. + # "production" = an engine can select this transport today (vLLM `--all2all-backend`, + # SGLang `--moe-a2a-backend`); "candidate" = a real transport we benchmark that no engine + # ships a selector for, so its numbers describe the library, not a deployable config. + # Mirrored in configs/platform_config.json; tests/test_matrix.py holds the two in step. maturity: str = "" SUPPORTED_MODES: tuple = ("normal",) # Dispatch precisions the adapter realizes. BF16 is the universal control; an diff --git a/experimental/CollectiveX/bench/ep_harness.py b/experimental/CollectiveX/bench/ep_harness.py index 71b58031fe..c52d30a70f 100644 --- a/experimental/CollectiveX/bench/ep_harness.py +++ b/experimental/CollectiveX/bench/ep_harness.py @@ -1131,9 +1131,7 @@ def run_sweep(args, backend, torch, dist, device, rank: int, world_size: int) -> # EPBackend.fp8_consume. Only meaningful when the case dispatches FP8. "fp8_consume": getattr(backend, "fp8_consume", None), "kernel_generation": kernel_generation(backend), - # Whether an inference engine can select this transport today — see - # EPBackend.maturity. A "candidate" row measures the library, not a - # deployable configuration, and must not be read as one. + # See EPBackend.maturity: a "candidate" row measures the library, not a deployment. "maturity": getattr(backend, "maturity", None) or "unknown", "name": backend.name, }, diff --git a/experimental/CollectiveX/docs/methodology.md b/experimental/CollectiveX/docs/methodology.md index 2a8840e8b0..371b5bb8da 100644 --- a/experimental/CollectiveX/docs/methodology.md +++ b/experimental/CollectiveX/docs/methodology.md @@ -87,12 +87,13 @@ cell-by-cell from the registry's `ll_backends` map rather than assumed wherever currently enabled for DeepEP V2 EP8 on H100/H200/B200, MoRI EP8 on MI300X/MI325X/MI355X, and UCCL-EP EP8 on H100/H200/B200 only (the legacy `Buffer` low-latency kernels over UCCL's CPU-proxy transport; the AMD SKUs keep UCCL-EP normal mode but drop LL, whose -kernel trips a warp-group assertion on AMD's CU count). NCCL EP implements the mode — its +kernel trips a warp-group assertion on AMD's CU count), and NCCL EP EP8 on all six NVIDIA SKUs — its `LOW_LATENCY` algorithm is the DeepEP-derived decode path, EXPERT_MAJOR receive with a source-side -weighted-kernel-sum combine — but carries no `ll_backends` row on any SKU: the shipped decode kernels -consume stale peer signals under a fixed workload and wedge -([NVIDIA/nccl#2303](https://github.com/NVIDIA/nccl/issues/2303)), so the cells stay out of the matrix -until a fixed wheel ships. Whether a given SKU/backend/EP/mode cell is attempted is a capability +weighted-kernel-sum combine. Those rows were dropped while every LL leg wedged on stale peer signals +([NVIDIA/nccl#2303](https://github.com/NVIDIA/nccl/issues/2303)) and restored once the single-handle +adapter removed the aliasing that caused it. B300, GB200 and GB300 carry NCCL EP as their only +low-latency row, and it is a `candidate` transport, so those three SKUs publish no production decode +coverage. Whether a given SKU/backend/EP/mode cell is attempted is a capability fact; whether it succeeded is decided only by the emitted artifact. ## Workload Identity diff --git a/experimental/CollectiveX/sweep_matrix.py b/experimental/CollectiveX/sweep_matrix.py index 9a3dbc0c11..00c3d5f30f 100644 --- a/experimental/CollectiveX/sweep_matrix.py +++ b/experimental/CollectiveX/sweep_matrix.py @@ -26,9 +26,7 @@ def _load_config(name: str) -> dict[str, Any]: SWEEP = _load_config("sweep.json") PLATFORMS = _load_config("platform_config.json")["platforms"] -# Whether an inference engine can select each transport today (see EPBackend.maturity). -# Consumed by the docs and the matrix; each adapter carries the same value for the -# artifact it writes, and tests/test_matrix.py holds the two in agreement. +# Per-backend production/candidate map for the matrix and docs; see EPBackend.maturity. BACKEND_MATURITY = _load_config("platform_config.json")["backend_maturity"] SWEEP_BACKENDS = tuple(dict.fromkeys( backend for platform in PLATFORMS.values() for backend in platform["backends"] diff --git a/experimental/CollectiveX/tests/test_matrix.py b/experimental/CollectiveX/tests/test_matrix.py index b799506b17..dc4becaada 100644 --- a/experimental/CollectiveX/tests/test_matrix.py +++ b/experimental/CollectiveX/tests/test_matrix.py @@ -282,13 +282,10 @@ def test_invalid_filters_fail_closed(self): class BackendMaturityTests(unittest.TestCase): - """The registry map and each adapter's `maturity` are two copies of one fact. - - The registry drives the matrix and the docs; the adapter attribute is what lands in - every case-attempt artifact. They are read by different consumers and can drift - silently, so pin both: complete coverage, a closed vocabulary, and agreement. The - adapter side is read from source rather than imported, because importing an adapter - pulls in torch and the vendor EP library, which the test image does not carry. + """The registry map and each adapter's `maturity` are two copies of one fact, read by + different consumers, so they can drift silently: pin coverage, vocabulary and agreement. + The adapter side is parsed from source rather than imported — importing an adapter pulls + in torch and the vendor EP library, which the test image does not carry. """ VOCABULARY = {"production", "candidate"}