[0z5a (0z5a)]

why

With 2 GPUs on Blackwell, ncclAllReduce through the MSCCL++ NCCL shim fails with ncclInvalidArgument for messages above 2 MiB when the call is captured in a CUDA graph. That is the usual setup for TP=2 inference with CUDA graphs.

• Selection: selectSingleNodeAllreduceBlackwell (src/ext/nccl/algorithm_selector.cc) returns default_allreduce_rsag_zero_copy when symmetric memory is off, the call is being captured, and the message is larger than 2 MiB. The Python default config (mscclpp_benchmark/comm.py:_default_tuned_config) does the same above 512 KiB.
• Kernel: AllreduceRsAgZeroCopyAdapter only instantiates the kernel for 4, 8, 16 and 32 ranks per IPC domain. For any other count it logs a warning and returns cudaErrorInvalidValue without launching anything.

Reproduced on 2× B300 (CUDA 13.3, driver 610.57) with nccl-tests all_reduce_perf -g 1 -c 1, using LD_PRELOAD=libmscclpp_nccl.so and no NCCL fallback library:

mode	16 KiB – 1 MiB	4 MiB and up
eager, bf16 / fp32	correct	correct up to 64 MiB
CUDA graph (-G 5), bf16 / fp32	correct	ncclInvalidArgument ("invalid argument")
plain NCCL, CUDA graph (control)	correct	correct up to 64 MiB

The Python benchmark (mscclpp_benchmark.bench_collective, 2 ranks, no --symmetric-memory) fails the same way, with "algorithm returned non-zero status" at every size of 640 KiB and above, for bf16, fp16 and fp8_e4m3fn.

what changed

The kernel is already generic in NRanks: NPeers = NRanks - 1, and the peer loops are unrolled at compile time. The setup path registers 2 * nPeers remote memories with no rank-count assumption. So this change adds the nRanksPerIpcDomain == 2 instantiation and updates the comment and the warning text.

Other rank counts (3, 5, 6, 7, …) are still rejected. If you'd prefer the selectors to check rank count and fall back to another algorithm for those, I can add that in this PR or in a follow-up.

verification

2× B300 (sm_103a build, CUDA 13.3), nccl-tests all_reduce_perf -b 16K -e 256M -f 4 -g 1 -c 1 -w 5 -n 20 with LD_PRELOAD=libmscclpp_nccl.so. The base is main at d5ce6c7. The table shows out-of-place time in µs, with busBW in GB/s in parentheses.

size	base, CUDA graph	this PR, CUDA graph	this PR, eager
1 MiB	12.74 (82)	12.95 (81)	19.30 (54)
4 MiB	invalid argument	18.91 (222)	31.00 (135)
16 MiB	—	38.86 (432)	53.15 (316)
64 MiB	—	113.13 (593)	148.58 (452)
256 MiB	—	421.67 (637)	537.31 (500)

• The table is for bf16. fp16 and fp32 behave the same: the base fails at 4 MiB under CUDA graph, and this PR passes every size.

• All runs with this PR report 0 wrong values.

• Eager-mode results are unchanged from the base, which is expected because that path does not use this algorithm.

• For reference, plain NCCL (CUDA graph, bf16) took 36.89 µs at 4 MiB, 50.94 µs at 16 MiB and 123.68 µs at 64 MiB. That was a separate run on a different GPU pair in the same node.

• clang-format --dry-run --Werror passes on the changed file (clang-format 21.1.6), and git diff --check passes.

🤖 Generated with Claude Code

[azure-pipelines]

Azure Pipelines:
There may be pipelines that require an authorized user to comment /azp run to run.

[Copilot]

Copilot was unable to review this pull request because the user who requested the review has insufficient remaining quota. Try requesting a review at a lower effort level.

[Binyang Li (Binyang2014)]

Thanks for the contribution, LGTM

[Copilot]

Copilot was unable to review this pull request because the user who requested the review has insufficient remaining quota. Try requesting a review at a lower effort level.