CollectiveX: label backend maturity and correct the low-latency docs / CollectiveX：标注后端成熟度并修正低延迟文档

experimental/CollectiveX/README.md

@@ -28,10 +28,12 @@ run in one of two modes:
   pure-intranode, same compact layout and unweighted rank-sum combine as `IntraNode`). It is a
   decode-phase-only, per-SKU-capability-gated addition whose runnable set differs from `normal`'s, so
   it is enabled from each SKU's `ll_backends` registry entry (currently DeepEP V2 EP8 on H100/H200/B200,
-  MoRI EP8 on MI300X/MI325X/MI355X, and UCCL-EP EP8 on H100/H200/B200 only — UCCL's low-latency kernel
-  trips a warp-group assertion on AMD's CU count, so the AMD SKUs keep UCCL-EP normal mode without LL;
-  NCCL EP has no low-latency row on any SKU while its decode kernels carry
-  [NVIDIA/nccl#2303](https://github.com/NVIDIA/nccl/issues/2303)).
+  MoRI EP8 on MI300X/MI325X/MI355X, UCCL-EP EP8 on H100/H200/B200 only — UCCL's low-latency kernel
+  trips a warp-group assertion on AMD's CU count, so the AMD SKUs keep UCCL-EP normal mode without LL —
+  and NCCL EP EP8 on all six NVIDIA SKUs, restored once the single-handle fix removed the
+  [NVIDIA/nccl#2303](https://github.com/NVIDIA/nccl/issues/2303) signal aliasing that had wedged them.
+  B300, GB200 and GB300 carry the `candidate` NCCL EP as their *only* low-latency row, so those
+  three SKUs have no production decode coverage; B300's DeepEP V2 LL row is unsupported, see below).
   Scoped single-node EP8 runs over the intra-node NVLink/XGMI
   low-latency path (no `/dev/gdrdrv` needed — validated on H200 with it absent); NVSHMEM/IBGDA on the
   wire is only a multi-node scale-out (EP16) concern.
@@ -67,12 +69,12 @@ frozen digest or locked case count.
 Physical host count does not determine scope: both GB topologies stay inside one 72-GPU MNNVL
 scale-up domain.
 
-| Backend | Current scope |
-|---|---|
-| DeepEP V2 | `normal` mode is PR #605 `ElasticBuffer` plus exact upstream #630 and #640 fixes: LSA for scale-up and GIN for x86 EP16 scale-out. FP8 dispatch via `use_fp8_dispatch` (blockwise e4m3fn) alongside BF16. `low-latency` mode is the legacy `deep_ep.Buffer` IBGDA decode kernels (per-expert padded layout, weighted combine, `use_fp8` e4m3fn), decode/EP8 only |
-| MoRI | `normal` mode uses the direct `IntraNode` kernel for scale-up EP8 on every CDNA SKU and pins `InterNodeV1` for EP16 over 2x8 XGMI + RDMA. `low-latency` mode selects the `IntraNodeLL` decode kernel (single-call, pure-intranode, same compact layout and unweighted combine as `IntraNode`), decode/EP8 only. FP8 dispatch is caller-prequantized (per-SKU e4m3fnuz on gfx942, e4m3fn on gfx950); combine stays BF16 (`quant_type=none`) alongside BF16 dispatch |
-| UCCL-EP | [UCCL](https://github.com/uccl-project/uccl) EP: a drop-in, API-identical DeepEP replacement whose CPU proxies issue GPUDirect RDMA over plain `libibverbs` (no NVSHMEM/IBGDA), with software message ordering, atomics, and flow control; scale-up is single-node `cudaIpc` over NVLink/XGMI (never MNNVL). `normal` mode is the legacy `Buffer` `dispatch`/`combine` (unweighted rank-sum); `low-latency` reuses the legacy `low_latency_dispatch`/`low_latency_combine` decode kernels (weighted combine), decode/EP8 only. FP8 dispatch is caller-prequantized in `normal` mode (blockwise e4m3fn, per-SKU e4m3fnuz on gfx942); in `low-latency` mode the caller sends BF16 and the decode kernel quantizes to e4m3 internally (`use_fp8`). Combine is BF16. Runs on NVIDIA and AMD (H100/H200/B200 + MI300X/MI325X/MI355X), EP8 scale-up. Cross-node EP16 is functional (the internode RDMA path connects and the light case passes correctness) but its CPU-proxy throughput overruns the standardized per-case wall-clock budget on heavy token counts, so EP16 is an unsupported coverage row for now |
-| NCCL EP | [NCCL EP](https://github.com/NVIDIA/nccl/tree/master/contrib/nccl_ep): NVIDIA's native MoE dispatch/combine on the NCCL Device API — LSA (NVLink load/store) intra-node, GIN (GPU-Initiated Networking) inter-node — driven through the `nccl4py` bindings. `normal` mode selects the `HIGH_THROUGHPUT` algorithm (FLAT `[N, hidden]` receive, unweighted rank-sum combine); the `LOW_LATENCY` algorithm is implemented in the adapter but has no enabled cell (see the `ll_backends` note above). BF16 only: NCCL EP's FP8 machinery exists upstream but its RELEASE.md lists it unsupported/untested, so no FP8 case is emitted. NVIDIA-only and CUDA 13 only. EP8 scale-up on H100/H200/B200/B300 plus EP8 and EP16 on GB200/GB300, where EP16 stays inside the MNNVL scale-up domain. x86 EP16 scale-out is an unsupported coverage row: the cross-node GIN path faults inside `nccl_ep.cc` identically on RoCE and IB across four SKUs, a GDAKI limit rather than a fabric-selection one |
+| Backend | Engine availability | Current scope |
+|---|---|---|
+| DeepEP V2 | `production` — vLLM `--all2all-backend deepep_v2`, SGLang `--moe-a2a-backend deepep` | `normal` mode is PR #605 `ElasticBuffer` plus exact upstream #630 and #640 fixes: LSA for scale-up and GIN for x86 EP16 scale-out. FP8 dispatch via `use_fp8_dispatch` (blockwise e4m3fn) alongside BF16. `low-latency` mode is the legacy `deep_ep.Buffer` IBGDA decode kernels (per-expert padded layout, weighted combine, `use_fp8` e4m3fn), decode/EP8 only. B300 is an unsupported coverage row in `low-latency`: the legacy Buffer self-enables NVSHMEM IBGDA even for a single-node EP8 run, and on B300 address-handle creation fails (`ibgda.cpp:2234 Unable to create ah`), rc255 on all eight ranks. `NVSHMEM_DISABLE_IB=1` does not help — the Buffer re-enables IBGDA regardless, and the run fails identically with it set and unset (measured on b300-002 and b300-011) |
+| MoRI | `production` — vLLM `--all2all-backend mori_*`, SGLang `--moe-a2a-backend mori` | `normal` mode uses the direct `IntraNode` kernel for scale-up EP8 on every CDNA SKU and pins `InterNodeV1` for EP16 over 2x8 XGMI + RDMA. `low-latency` mode selects the `IntraNodeLL` decode kernel (single-call, pure-intranode, same compact layout and unweighted combine as `IntraNode`), decode/EP8 only. FP8 dispatch is caller-prequantized (per-SKU e4m3fnuz on gfx942, e4m3fn on gfx950); combine stays BF16 (`quant_type=none`) alongside BF16 dispatch |
+| UCCL-EP | `candidate` — no engine exposes a UCCL-EP selector | [UCCL](https://github.com/uccl-project/uccl) EP: a drop-in, API-identical DeepEP replacement whose CPU proxies issue GPUDirect RDMA over plain `libibverbs` (no NVSHMEM/IBGDA), with software message ordering, atomics, and flow control; scale-up is single-node `cudaIpc` over NVLink/XGMI (never MNNVL). `normal` mode is the legacy `Buffer` `dispatch`/`combine` (unweighted rank-sum); `low-latency` reuses the legacy `low_latency_dispatch`/`low_latency_combine` decode kernels (weighted combine), decode/EP8 only. FP8 dispatch is caller-prequantized in `normal` mode (blockwise e4m3fn, per-SKU e4m3fnuz on gfx942); in `low-latency` mode the caller sends BF16 and the decode kernel quantizes to e4m3 internally (`use_fp8`). Combine is BF16. Runs on NVIDIA and AMD (H100/H200/B200 + MI300X/MI325X/MI355X), EP8 scale-up. Cross-node EP16 is functional (the internode RDMA path connects and the light case passes correctness) but its CPU-proxy throughput overruns the standardized per-case wall-clock budget on heavy token counts, so EP16 is an unsupported coverage row for now |
+| NCCL EP | `candidate` — NVIDIA's own library, but no engine exposes an NCCL-EP selector | [NCCL EP](https://github.com/NVIDIA/nccl/tree/master/contrib/nccl_ep): NVIDIA's native MoE dispatch/combine on the NCCL Device API — LSA (NVLink load/store) intra-node, GIN (GPU-Initiated Networking) inter-node — driven through the `nccl4py` bindings. `normal` mode selects the `HIGH_THROUGHPUT` algorithm (FLAT `[N, hidden]` receive, unweighted rank-sum combine); the `LOW_LATENCY` algorithm carries an EP8 `ll_backends` row on all six NVIDIA SKUs, restored once the single-handle fix removed the NVIDIA/nccl#2303 signal aliasing. BF16 only: NCCL EP's FP8 machinery exists upstream but its RELEASE.md lists it unsupported/untested, so no FP8 case is emitted. NVIDIA-only and CUDA 13 only. EP8 scale-up on H100/H200/B200/B300 plus EP8 and EP16 on GB200/GB300, where EP16 stays inside the MNNVL scale-up domain. x86 EP16 scale-out is an unsupported coverage row: the cross-node GIN path faults inside `nccl_ep.cc` identically on RoCE and IB across four SKUs, a GDAKI limit rather than a fabric-selection one |
 
 DeepEP V2 means the `ElasticBuffer` implementation introduced by
 [DeepEP PR #605](https://github.com/deepseek-ai/DeepEP/pull/605), not a newer legacy `Buffer` build.

experimental/CollectiveX/bench/ep_backend.py

@@ -67,6 +67,11 @@ class EPBackend(abc.ABC):
     """
 
     name: str = ""
+    # "production" = an engine can select this transport today (vLLM `--all2all-backend`,
+    # SGLang `--moe-a2a-backend`); "candidate" = a real transport we benchmark that no engine
+    # ships a selector for, so its numbers describe the library, not a deployable config.
+    # Mirrored in configs/platform_config.json; tests/test_matrix.py holds the two in step.
+    maturity: str = ""
     SUPPORTED_MODES: tuple = ("normal",)
     # Dispatch precisions the adapter realizes. BF16 is the universal control; an
     # adapter that also sends an FP8-quantized dispatch payload widens this.

experimental/CollectiveX/bench/ep_deepep_v2.py

@@ -116,6 +116,7 @@ def _require_runtime() -> None:
 
 class DeepEPV2Backend(EPBackend):
     name = "deepep-v2"
+    maturity = "production"  # vLLM --all2all-backend deepep_v2; SGLang --moe-a2a-backend deepep
     # Two kernel families under one adapter, selected by mode:
     #   normal      -> PR #605 ElasticBuffer (LSA vs hybrid GIN are transport paths, not
     #                  kernel families); rank-deduplicated unweighted-rank-sum combine.

experimental/CollectiveX/bench/ep_harness.py

@@ -1131,6 +1131,8 @@ def run_sweep(args, backend, torch, dist, device, rank: int, world_size: int) ->
             # EPBackend.fp8_consume. Only meaningful when the case dispatches FP8.
             "fp8_consume": getattr(backend, "fp8_consume", None),
             "kernel_generation": kernel_generation(backend),
+            # See EPBackend.maturity: a "candidate" row measures the library, not a deployment.
+            "maturity": getattr(backend, "maturity", None) or "unknown",
             "name": backend.name,
         },
         "topology": {

experimental/CollectiveX/bench/ep_mori.py

@@ -38,6 +38,7 @@ def _project_local_metadata(torch_module, raw_expert_ids, raw_weights, rank, exp
 
 class MoRIBackend(EPBackend):
     name = "mori"
+    maturity = "production"  # vLLM --all2all-backend mori_*; SGLang --moe-a2a-backend mori
     SUPPORTED_MODES = ("normal", "low-latency")
     SUPPORTED_PRECISIONS = ("bf16", "fp8")
     combine_needs_redispatch = True

experimental/CollectiveX/bench/ep_nccl.py

@@ -65,6 +65,7 @@
 
 class NCCLEPBackend(EPBackend):
     name = "nccl-ep"
+    maturity = "candidate"  # NVIDIA's library, but no engine exposes an NCCL-EP selector
     # One library, two algorithms selected by args.mode. kernel_generation and the combine
     # semantics are switched to their LL values in __init__ (mirrors ep_deepep_v2).
     #   normal      -> HT / FLAT layout / unweighted-rank-sum combine.

experimental/CollectiveX/bench/ep_uccl.py

@@ -123,6 +123,7 @@ def _normal_num_sms() -> int:
 
 class UCCLEPBackend(EPBackend):
     name = "uccl-ep"
+    maturity = "candidate"  # no engine exposes a UCCL-EP all-to-all selector
     # One legacy Buffer under two modes, selected by args.mode:
     #   normal      -> get_dispatch_layout/dispatch/combine; unweighted rank-sum combine.
     #   low-latency -> low_latency_dispatch/combine decode kernels; source-side weighted combine.

experimental/CollectiveX/configs/platform_config.json

@@ -1,4 +1,5 @@
 {
+  "backend_maturity": {"deepep-v2": "production", "mori": "production", "uccl-ep": "candidate", "nccl-ep": "candidate"},
   "platforms": {
     "h100-dgxc": {
       "arch": "sm90",

experimental/CollectiveX/docs/methodology.md

@@ -87,12 +87,13 @@ cell-by-cell from the registry's `ll_backends` map rather than assumed wherever
 currently enabled for DeepEP V2 EP8 on H100/H200/B200, MoRI
 EP8 on MI300X/MI325X/MI355X, and UCCL-EP EP8 on H100/H200/B200 only (the legacy `Buffer` low-latency
 kernels over UCCL's CPU-proxy transport; the AMD SKUs keep UCCL-EP normal mode but drop LL, whose
-kernel trips a warp-group assertion on AMD's CU count). NCCL EP implements the mode — its
+kernel trips a warp-group assertion on AMD's CU count), and NCCL EP EP8 on all six NVIDIA SKUs — its
 `LOW_LATENCY` algorithm is the DeepEP-derived decode path, EXPERT_MAJOR receive with a source-side
-weighted-kernel-sum combine — but carries no `ll_backends` row on any SKU: the shipped decode kernels
-consume stale peer signals under a fixed workload and wedge
-([NVIDIA/nccl#2303](https://github.com/NVIDIA/nccl/issues/2303)), so the cells stay out of the matrix
-until a fixed wheel ships. Whether a given SKU/backend/EP/mode cell is attempted is a capability
+weighted-kernel-sum combine. Those rows were dropped while every LL leg wedged on stale peer signals
+([NVIDIA/nccl#2303](https://github.com/NVIDIA/nccl/issues/2303)) and restored once the single-handle
+adapter removed the aliasing that caused it. B300, GB200 and GB300 carry NCCL EP as their only
+low-latency row, and it is a `candidate` transport, so those three SKUs publish no production decode
+coverage. Whether a given SKU/backend/EP/mode cell is attempted is a capability
 fact; whether it succeeded is decided only by the emitted artifact.
 
 ## Workload Identity
@@ -202,7 +203,11 @@ One raw case document carries `record_type: "case-attempt"` and the single `vers
 - `workload`: `cross_rank_consistent`, whether the routing trace was proven identical across ranks;
 - `measurement`: dispatch/combine dtype (the realized wire formats — combine always BF16, dispatch
   BF16 or the SKU's FP8 format) and semantics, `sampling`, and the per-point `rows`;
-- `implementation`: backend name and kernel generation;
+- `implementation`: backend name, kernel generation, and `maturity` — whether a production
+  inference engine can select this transport today (`production` = exposed by vLLM's
+  `--all2all-backend` or SGLang's `--moe-a2a-backend`; `candidate` = a real transport we
+  benchmark that no engine ships a selector for, so its numbers describe the library rather
+  than a deployable configuration). The same map is in the registry's `backend_maturity`;
 - `topology`: requested SKU/product, placement, nodes, scale-up domain, transport, and world size;
 - `provenance`: the mounted image tag and source SHA; and
 - `outcome`: `status` (`success` or `invalid`) and `reasons`.

experimental/CollectiveX/sweep_matrix.py

@@ -26,6 +26,8 @@ def _load_config(name: str) -> dict[str, Any]:
 
 SWEEP = _load_config("sweep.json")
 PLATFORMS = _load_config("platform_config.json")["platforms"]
+# Per-backend production/candidate map for the matrix and docs; see EPBackend.maturity.
+BACKEND_MATURITY = _load_config("platform_config.json")["backend_maturity"]
 SWEEP_BACKENDS = tuple(dict.fromkeys(
     backend for platform in PLATFORMS.values() for backend in platform["backends"]
 ))

experimental/CollectiveX/tests/test_matrix.py

@@ -281,5 +281,57 @@ def test_invalid_filters_fail_closed(self):
                 sweep_matrix.resolve_matrix(**options)
 
 
+class BackendMaturityTests(unittest.TestCase):
+    """The registry map and each adapter's `maturity` are two copies of one fact, read by
+    different consumers, so they can drift silently: pin coverage, vocabulary and agreement.
+    The adapter side is parsed from source rather than imported — importing an adapter pulls
+    in torch and the vendor EP library, which the test image does not carry.
+    """
+
+    VOCABULARY = {"production", "candidate"}
+
+    @staticmethod
+    def _declared_in_source():
+        """{backend name: maturity} parsed from the adapter class bodies."""
+        import ast
+
+        declared = {}
+        for path in sorted((ROOT / "bench").glob("ep_*.py")):
+            tree = ast.parse(path.read_text())
+            for node in ast.walk(tree):
+                if not isinstance(node, ast.ClassDef):
+                    continue
+                literals = {}
+                for statement in node.body:
+                    if not isinstance(statement, ast.Assign):
+                        continue
+                    if not isinstance(statement.value, ast.Constant):
+                        continue
+                    for target in statement.targets:
+                        if isinstance(target, ast.Name):
+                            literals[target.id] = statement.value.value
+                # The abstract base declares both as empty defaults; skip it.
+                if literals.get("name") and "maturity" in literals:
+                    declared[literals["name"]] = literals["maturity"]
+        return declared
+
+    def test_registry_covers_every_dispatched_backend(self):
+        maturity = sweep_matrix.BACKEND_MATURITY
+        for sku, platform in sweep_matrix.PLATFORMS.items():
+            for backend in platform["backends"]:
+                with self.subTest(sku=sku, backend=backend):
+                    self.assertIn(backend, maturity)
+                    self.assertIn(maturity[backend], self.VOCABULARY)
+
+    def test_adapters_and_registry_agree(self):
+        declared = self._declared_in_source()
+        # Every backend the matrix can dispatch must declare a maturity in its adapter,
+        # or the artifact it writes would say "unknown" while the registry says otherwise.
+        for backend, expected in sweep_matrix.BACKEND_MATURITY.items():
+            with self.subTest(backend=backend):
+                self.assertIn(backend, declared)
+                self.assertEqual(declared[backend], expected)
+
+
 if __name__ == "__main__":
     unittest.main()