diff --git a/experimental/CollectiveX/README.md b/experimental/CollectiveX/README.md index 1e1f0ce521..956e741d98 100644 --- a/experimental/CollectiveX/README.md +++ b/experimental/CollectiveX/README.md @@ -28,10 +28,12 @@ run in one of two modes: pure-intranode, same compact layout and unweighted rank-sum combine as `IntraNode`). It is a decode-phase-only, per-SKU-capability-gated addition whose runnable set differs from `normal`'s, so it is enabled from each SKU's `ll_backends` registry entry (currently DeepEP V2 EP8 on H100/H200/B200, - MoRI EP8 on MI300X/MI325X/MI355X, and UCCL-EP EP8 on H100/H200/B200 only — UCCL's low-latency kernel - trips a warp-group assertion on AMD's CU count, so the AMD SKUs keep UCCL-EP normal mode without LL; - NCCL EP has no low-latency row on any SKU while its decode kernels carry - [NVIDIA/nccl#2303](https://github.com/NVIDIA/nccl/issues/2303)). + MoRI EP8 on MI300X/MI325X/MI355X, UCCL-EP EP8 on H100/H200/B200 only — UCCL's low-latency kernel + trips a warp-group assertion on AMD's CU count, so the AMD SKUs keep UCCL-EP normal mode without LL — + and NCCL EP EP8 on all six NVIDIA SKUs, restored once the single-handle fix removed the + [NVIDIA/nccl#2303](https://github.com/NVIDIA/nccl/issues/2303) signal aliasing that had wedged them. + B300, GB200 and GB300 carry the `candidate` NCCL EP as their *only* low-latency row, so those + three SKUs have no production decode coverage; B300's DeepEP V2 LL row is unsupported, see below). Scoped single-node EP8 runs over the intra-node NVLink/XGMI low-latency path (no `/dev/gdrdrv` needed — validated on H200 with it absent); NVSHMEM/IBGDA on the wire is only a multi-node scale-out (EP16) concern. @@ -67,12 +69,12 @@ frozen digest or locked case count. Physical host count does not determine scope: both GB topologies stay inside one 72-GPU MNNVL scale-up domain. -| Backend | Current scope | -|---|---| -| DeepEP V2 | `normal` mode is PR #605 `ElasticBuffer` plus exact upstream #630 and #640 fixes: LSA for scale-up and GIN for x86 EP16 scale-out. FP8 dispatch via `use_fp8_dispatch` (blockwise e4m3fn) alongside BF16. `low-latency` mode is the legacy `deep_ep.Buffer` IBGDA decode kernels (per-expert padded layout, weighted combine, `use_fp8` e4m3fn), decode/EP8 only | -| MoRI | `normal` mode uses the direct `IntraNode` kernel for scale-up EP8 on every CDNA SKU and pins `InterNodeV1` for EP16 over 2x8 XGMI + RDMA. `low-latency` mode selects the `IntraNodeLL` decode kernel (single-call, pure-intranode, same compact layout and unweighted combine as `IntraNode`), decode/EP8 only. FP8 dispatch is caller-prequantized (per-SKU e4m3fnuz on gfx942, e4m3fn on gfx950); combine stays BF16 (`quant_type=none`) alongside BF16 dispatch | -| UCCL-EP | [UCCL](https://github.com/uccl-project/uccl) EP: a drop-in, API-identical DeepEP replacement whose CPU proxies issue GPUDirect RDMA over plain `libibverbs` (no NVSHMEM/IBGDA), with software message ordering, atomics, and flow control; scale-up is single-node `cudaIpc` over NVLink/XGMI (never MNNVL). `normal` mode is the legacy `Buffer` `dispatch`/`combine` (unweighted rank-sum); `low-latency` reuses the legacy `low_latency_dispatch`/`low_latency_combine` decode kernels (weighted combine), decode/EP8 only. FP8 dispatch is caller-prequantized in `normal` mode (blockwise e4m3fn, per-SKU e4m3fnuz on gfx942); in `low-latency` mode the caller sends BF16 and the decode kernel quantizes to e4m3 internally (`use_fp8`). Combine is BF16. Runs on NVIDIA and AMD (H100/H200/B200 + MI300X/MI325X/MI355X), EP8 scale-up. Cross-node EP16 is functional (the internode RDMA path connects and the light case passes correctness) but its CPU-proxy throughput overruns the standardized per-case wall-clock budget on heavy token counts, so EP16 is an unsupported coverage row for now | -| NCCL EP | [NCCL EP](https://github.com/NVIDIA/nccl/tree/master/contrib/nccl_ep): NVIDIA's native MoE dispatch/combine on the NCCL Device API — LSA (NVLink load/store) intra-node, GIN (GPU-Initiated Networking) inter-node — driven through the `nccl4py` bindings. `normal` mode selects the `HIGH_THROUGHPUT` algorithm (FLAT `[N, hidden]` receive, unweighted rank-sum combine); the `LOW_LATENCY` algorithm is implemented in the adapter but has no enabled cell (see the `ll_backends` note above). BF16 only: NCCL EP's FP8 machinery exists upstream but its RELEASE.md lists it unsupported/untested, so no FP8 case is emitted. NVIDIA-only and CUDA 13 only. EP8 scale-up on H100/H200/B200/B300 plus EP8 and EP16 on GB200/GB300, where EP16 stays inside the MNNVL scale-up domain. x86 EP16 scale-out is an unsupported coverage row: the cross-node GIN path faults inside `nccl_ep.cc` identically on RoCE and IB across four SKUs, a GDAKI limit rather than a fabric-selection one | +| Backend | Engine availability | Current scope | +|---|---|---| +| DeepEP V2 | `production` — vLLM `--all2all-backend deepep_v2`, SGLang `--moe-a2a-backend deepep` | `normal` mode is PR #605 `ElasticBuffer` plus exact upstream #630 and #640 fixes: LSA for scale-up and GIN for x86 EP16 scale-out. FP8 dispatch via `use_fp8_dispatch` (blockwise e4m3fn) alongside BF16. `low-latency` mode is the legacy `deep_ep.Buffer` IBGDA decode kernels (per-expert padded layout, weighted combine, `use_fp8` e4m3fn), decode/EP8 only. B300 is an unsupported coverage row in `low-latency`: the legacy Buffer self-enables NVSHMEM IBGDA even for a single-node EP8 run, and on B300 address-handle creation fails (`ibgda.cpp:2234 Unable to create ah`), rc255 on all eight ranks. `NVSHMEM_DISABLE_IB=1` does not help — the Buffer re-enables IBGDA regardless, and the run fails identically with it set and unset (measured on b300-002 and b300-011) | +| MoRI | `production` — vLLM `--all2all-backend mori_*`, SGLang `--moe-a2a-backend mori` | `normal` mode uses the direct `IntraNode` kernel for scale-up EP8 on every CDNA SKU and pins `InterNodeV1` for EP16 over 2x8 XGMI + RDMA. `low-latency` mode selects the `IntraNodeLL` decode kernel (single-call, pure-intranode, same compact layout and unweighted combine as `IntraNode`), decode/EP8 only. FP8 dispatch is caller-prequantized (per-SKU e4m3fnuz on gfx942, e4m3fn on gfx950); combine stays BF16 (`quant_type=none`) alongside BF16 dispatch | +| UCCL-EP | `candidate` — no engine exposes a UCCL-EP selector | [UCCL](https://github.com/uccl-project/uccl) EP: a drop-in, API-identical DeepEP replacement whose CPU proxies issue GPUDirect RDMA over plain `libibverbs` (no NVSHMEM/IBGDA), with software message ordering, atomics, and flow control; scale-up is single-node `cudaIpc` over NVLink/XGMI (never MNNVL). `normal` mode is the legacy `Buffer` `dispatch`/`combine` (unweighted rank-sum); `low-latency` reuses the legacy `low_latency_dispatch`/`low_latency_combine` decode kernels (weighted combine), decode/EP8 only. FP8 dispatch is caller-prequantized in `normal` mode (blockwise e4m3fn, per-SKU e4m3fnuz on gfx942); in `low-latency` mode the caller sends BF16 and the decode kernel quantizes to e4m3 internally (`use_fp8`). Combine is BF16. Runs on NVIDIA and AMD (H100/H200/B200 + MI300X/MI325X/MI355X), EP8 scale-up. Cross-node EP16 is functional (the internode RDMA path connects and the light case passes correctness) but its CPU-proxy throughput overruns the standardized per-case wall-clock budget on heavy token counts, so EP16 is an unsupported coverage row for now | +| NCCL EP | `candidate` — NVIDIA's own library, but no engine exposes an NCCL-EP selector | [NCCL EP](https://github.com/NVIDIA/nccl/tree/master/contrib/nccl_ep): NVIDIA's native MoE dispatch/combine on the NCCL Device API — LSA (NVLink load/store) intra-node, GIN (GPU-Initiated Networking) inter-node — driven through the `nccl4py` bindings. `normal` mode selects the `HIGH_THROUGHPUT` algorithm (FLAT `[N, hidden]` receive, unweighted rank-sum combine); the `LOW_LATENCY` algorithm carries an EP8 `ll_backends` row on all six NVIDIA SKUs, restored once the single-handle fix removed the NVIDIA/nccl#2303 signal aliasing. BF16 only: NCCL EP's FP8 machinery exists upstream but its RELEASE.md lists it unsupported/untested, so no FP8 case is emitted. NVIDIA-only and CUDA 13 only. EP8 scale-up on H100/H200/B200/B300 plus EP8 and EP16 on GB200/GB300, where EP16 stays inside the MNNVL scale-up domain. x86 EP16 scale-out is an unsupported coverage row: the cross-node GIN path faults inside `nccl_ep.cc` identically on RoCE and IB across four SKUs, a GDAKI limit rather than a fabric-selection one | DeepEP V2 means the `ElasticBuffer` implementation introduced by [DeepEP PR #605](https://github.com/deepseek-ai/DeepEP/pull/605), not a newer legacy `Buffer` build. diff --git a/experimental/CollectiveX/bench/ep_backend.py b/experimental/CollectiveX/bench/ep_backend.py index faf2726e15..a16d1a015b 100644 --- a/experimental/CollectiveX/bench/ep_backend.py +++ b/experimental/CollectiveX/bench/ep_backend.py @@ -67,6 +67,11 @@ class EPBackend(abc.ABC): """ name: str = "" + # "production" = an engine can select this transport today (vLLM `--all2all-backend`, + # SGLang `--moe-a2a-backend`); "candidate" = a real transport we benchmark that no engine + # ships a selector for, so its numbers describe the library, not a deployable config. + # Mirrored in configs/platform_config.json; tests/test_matrix.py holds the two in step. + maturity: str = "" SUPPORTED_MODES: tuple = ("normal",) # Dispatch precisions the adapter realizes. BF16 is the universal control; an # adapter that also sends an FP8-quantized dispatch payload widens this. diff --git a/experimental/CollectiveX/bench/ep_deepep_v2.py b/experimental/CollectiveX/bench/ep_deepep_v2.py index 8708d9886d..f74cfcf873 100644 --- a/experimental/CollectiveX/bench/ep_deepep_v2.py +++ b/experimental/CollectiveX/bench/ep_deepep_v2.py @@ -116,6 +116,7 @@ def _require_runtime() -> None: class DeepEPV2Backend(EPBackend): name = "deepep-v2" + maturity = "production" # vLLM --all2all-backend deepep_v2; SGLang --moe-a2a-backend deepep # Two kernel families under one adapter, selected by mode: # normal -> PR #605 ElasticBuffer (LSA vs hybrid GIN are transport paths, not # kernel families); rank-deduplicated unweighted-rank-sum combine. diff --git a/experimental/CollectiveX/bench/ep_harness.py b/experimental/CollectiveX/bench/ep_harness.py index 387d4a44a4..c52d30a70f 100644 --- a/experimental/CollectiveX/bench/ep_harness.py +++ b/experimental/CollectiveX/bench/ep_harness.py @@ -1131,6 +1131,8 @@ def run_sweep(args, backend, torch, dist, device, rank: int, world_size: int) -> # EPBackend.fp8_consume. Only meaningful when the case dispatches FP8. "fp8_consume": getattr(backend, "fp8_consume", None), "kernel_generation": kernel_generation(backend), + # See EPBackend.maturity: a "candidate" row measures the library, not a deployment. + "maturity": getattr(backend, "maturity", None) or "unknown", "name": backend.name, }, "topology": { diff --git a/experimental/CollectiveX/bench/ep_mori.py b/experimental/CollectiveX/bench/ep_mori.py index e3adeeb71b..2cda3f974c 100644 --- a/experimental/CollectiveX/bench/ep_mori.py +++ b/experimental/CollectiveX/bench/ep_mori.py @@ -38,6 +38,7 @@ def _project_local_metadata(torch_module, raw_expert_ids, raw_weights, rank, exp class MoRIBackend(EPBackend): name = "mori" + maturity = "production" # vLLM --all2all-backend mori_*; SGLang --moe-a2a-backend mori SUPPORTED_MODES = ("normal", "low-latency") SUPPORTED_PRECISIONS = ("bf16", "fp8") combine_needs_redispatch = True diff --git a/experimental/CollectiveX/bench/ep_nccl.py b/experimental/CollectiveX/bench/ep_nccl.py index 14b4c2de9e..3b1010fe7b 100644 --- a/experimental/CollectiveX/bench/ep_nccl.py +++ b/experimental/CollectiveX/bench/ep_nccl.py @@ -65,6 +65,7 @@ class NCCLEPBackend(EPBackend): name = "nccl-ep" + maturity = "candidate" # NVIDIA's library, but no engine exposes an NCCL-EP selector # One library, two algorithms selected by args.mode. kernel_generation and the combine # semantics are switched to their LL values in __init__ (mirrors ep_deepep_v2). # normal -> HT / FLAT layout / unweighted-rank-sum combine. diff --git a/experimental/CollectiveX/bench/ep_uccl.py b/experimental/CollectiveX/bench/ep_uccl.py index 821b23c25d..9ef1fa7899 100644 --- a/experimental/CollectiveX/bench/ep_uccl.py +++ b/experimental/CollectiveX/bench/ep_uccl.py @@ -123,6 +123,7 @@ def _normal_num_sms() -> int: class UCCLEPBackend(EPBackend): name = "uccl-ep" + maturity = "candidate" # no engine exposes a UCCL-EP all-to-all selector # One legacy Buffer under two modes, selected by args.mode: # normal -> get_dispatch_layout/dispatch/combine; unweighted rank-sum combine. # low-latency -> low_latency_dispatch/combine decode kernels; source-side weighted combine. diff --git a/experimental/CollectiveX/configs/platform_config.json b/experimental/CollectiveX/configs/platform_config.json index 3babb66536..018c79805a 100644 --- a/experimental/CollectiveX/configs/platform_config.json +++ b/experimental/CollectiveX/configs/platform_config.json @@ -1,4 +1,5 @@ { + "backend_maturity": {"deepep-v2": "production", "mori": "production", "uccl-ep": "candidate", "nccl-ep": "candidate"}, "platforms": { "h100-dgxc": { "arch": "sm90", diff --git a/experimental/CollectiveX/docs/methodology.md b/experimental/CollectiveX/docs/methodology.md index 4ec82c7258..371b5bb8da 100644 --- a/experimental/CollectiveX/docs/methodology.md +++ b/experimental/CollectiveX/docs/methodology.md @@ -87,12 +87,13 @@ cell-by-cell from the registry's `ll_backends` map rather than assumed wherever currently enabled for DeepEP V2 EP8 on H100/H200/B200, MoRI EP8 on MI300X/MI325X/MI355X, and UCCL-EP EP8 on H100/H200/B200 only (the legacy `Buffer` low-latency kernels over UCCL's CPU-proxy transport; the AMD SKUs keep UCCL-EP normal mode but drop LL, whose -kernel trips a warp-group assertion on AMD's CU count). NCCL EP implements the mode — its +kernel trips a warp-group assertion on AMD's CU count), and NCCL EP EP8 on all six NVIDIA SKUs — its `LOW_LATENCY` algorithm is the DeepEP-derived decode path, EXPERT_MAJOR receive with a source-side -weighted-kernel-sum combine — but carries no `ll_backends` row on any SKU: the shipped decode kernels -consume stale peer signals under a fixed workload and wedge -([NVIDIA/nccl#2303](https://github.com/NVIDIA/nccl/issues/2303)), so the cells stay out of the matrix -until a fixed wheel ships. Whether a given SKU/backend/EP/mode cell is attempted is a capability +weighted-kernel-sum combine. Those rows were dropped while every LL leg wedged on stale peer signals +([NVIDIA/nccl#2303](https://github.com/NVIDIA/nccl/issues/2303)) and restored once the single-handle +adapter removed the aliasing that caused it. B300, GB200 and GB300 carry NCCL EP as their only +low-latency row, and it is a `candidate` transport, so those three SKUs publish no production decode +coverage. Whether a given SKU/backend/EP/mode cell is attempted is a capability fact; whether it succeeded is decided only by the emitted artifact. ## Workload Identity @@ -202,7 +203,11 @@ One raw case document carries `record_type: "case-attempt"` and the single `vers - `workload`: `cross_rank_consistent`, whether the routing trace was proven identical across ranks; - `measurement`: dispatch/combine dtype (the realized wire formats — combine always BF16, dispatch BF16 or the SKU's FP8 format) and semantics, `sampling`, and the per-point `rows`; -- `implementation`: backend name and kernel generation; +- `implementation`: backend name, kernel generation, and `maturity` — whether a production + inference engine can select this transport today (`production` = exposed by vLLM's + `--all2all-backend` or SGLang's `--moe-a2a-backend`; `candidate` = a real transport we + benchmark that no engine ships a selector for, so its numbers describe the library rather + than a deployable configuration). The same map is in the registry's `backend_maturity`; - `topology`: requested SKU/product, placement, nodes, scale-up domain, transport, and world size; - `provenance`: the mounted image tag and source SHA; and - `outcome`: `status` (`success` or `invalid`) and `reasons`. diff --git a/experimental/CollectiveX/sweep_matrix.py b/experimental/CollectiveX/sweep_matrix.py index 9a38f7f9ee..00c3d5f30f 100644 --- a/experimental/CollectiveX/sweep_matrix.py +++ b/experimental/CollectiveX/sweep_matrix.py @@ -26,6 +26,8 @@ def _load_config(name: str) -> dict[str, Any]: SWEEP = _load_config("sweep.json") PLATFORMS = _load_config("platform_config.json")["platforms"] +# Per-backend production/candidate map for the matrix and docs; see EPBackend.maturity. +BACKEND_MATURITY = _load_config("platform_config.json")["backend_maturity"] SWEEP_BACKENDS = tuple(dict.fromkeys( backend for platform in PLATFORMS.values() for backend in platform["backends"] )) diff --git a/experimental/CollectiveX/tests/test_matrix.py b/experimental/CollectiveX/tests/test_matrix.py index 83d3c2a4b7..dc4becaada 100644 --- a/experimental/CollectiveX/tests/test_matrix.py +++ b/experimental/CollectiveX/tests/test_matrix.py @@ -281,5 +281,57 @@ def test_invalid_filters_fail_closed(self): sweep_matrix.resolve_matrix(**options) +class BackendMaturityTests(unittest.TestCase): + """The registry map and each adapter's `maturity` are two copies of one fact, read by + different consumers, so they can drift silently: pin coverage, vocabulary and agreement. + The adapter side is parsed from source rather than imported — importing an adapter pulls + in torch and the vendor EP library, which the test image does not carry. + """ + + VOCABULARY = {"production", "candidate"} + + @staticmethod + def _declared_in_source(): + """{backend name: maturity} parsed from the adapter class bodies.""" + import ast + + declared = {} + for path in sorted((ROOT / "bench").glob("ep_*.py")): + tree = ast.parse(path.read_text()) + for node in ast.walk(tree): + if not isinstance(node, ast.ClassDef): + continue + literals = {} + for statement in node.body: + if not isinstance(statement, ast.Assign): + continue + if not isinstance(statement.value, ast.Constant): + continue + for target in statement.targets: + if isinstance(target, ast.Name): + literals[target.id] = statement.value.value + # The abstract base declares both as empty defaults; skip it. + if literals.get("name") and "maturity" in literals: + declared[literals["name"]] = literals["maturity"] + return declared + + def test_registry_covers_every_dispatched_backend(self): + maturity = sweep_matrix.BACKEND_MATURITY + for sku, platform in sweep_matrix.PLATFORMS.items(): + for backend in platform["backends"]: + with self.subTest(sku=sku, backend=backend): + self.assertIn(backend, maturity) + self.assertIn(maturity[backend], self.VOCABULARY) + + def test_adapters_and_registry_agree(self): + declared = self._declared_in_source() + # Every backend the matrix can dispatch must declare a maturity in its adapter, + # or the artifact it writes would say "unknown" while the registry says otherwise. + for backend, expected in sweep_matrix.BACKEND_MATURITY.items(): + with self.subTest(backend=backend): + self.assertIn(backend, declared) + self.assertEqual(declared[backend], expected) + + if __name__ == "__main__": unittest.main()