Repository navigation
PortChannel: add optional GPUNetIO backend with pinned FetchContent dependency - #910
Qinghua Zhou (seagater) wants to merge 72 commits into
Conversation
Port the generic integration and safety fixes from qinghuazhou/gpunetio_port_channel_merge_feature_ep onto main, preserving CPU-proxy defaults and accumulate semantics. Exclude all EP kernels and policy. Pin the unmodified NVIDIA-DOCA/gpunetio v4.0.1 submodule. Add public host channel construction, CPU regressions, two-rank smoke coverage and installation instructions. ON/OFF CUDA builds, six CPU checks, installed-header compilation and lint pass. GPU/NIC runtime remains unverified; documented upstream allocation and CPU-service shutdown issues block merging.
Port GpuNetIoMultiQpBandwidth from the EP integration branch with configurable single-HCA QPs, collective geometry validation, queue-indexed device operations and per-QP atomic scratch. Preserve single-QP defaults and add CPU queue-isolation regression coverage. Validated GPUNetIO ON/OFF builds, seven CPU test runs and repository lint. GPU/NIC benchmark execution remains pending; the upstream submodule is unchanged.
|
Azure Pipelines: There may be pipelines that require an authorized user to comment /azp run to run. |
Replace the dense-only upstream flatten helper with a checked sparse table adapter that keeps null self-rank slots zero and preserves remote peer/QP indices. Own the uploaded CUDA table in GpuNetIoService without modifying the upstream submodule. Add actual-upstream-type regression coverage for 28 layouts and 296 rejected inputs. Commit the exact patch verified by the user; ON/OFF builds, CPU regressions and lint were validated separately.
There was a problem hiding this comment.
Copilot review overview
🔵 Needs a closer look
Device completion and peer-validation defects remain alongside acknowledged upstream blockers and unverified GPU/NIC runtime behavior.
Review effort: Balanced
Findings: 1
Open (7)
Reject self and out-of-range peer ranks during construction · New Validate peer before indexing peer arrays · New Release builds discard EBUSY instead of completing bounded flush · New Avoid enabling backend with broken pinned upstream implementation · New Add required MIT license header to public C++ header · New Add required MIT license header to C++ header · New Add required MIT license header to C++ test file · New
What changed in this PR
Adds an optional GPUNetIO/GDAKI backend to PortChannel while retaining CPU proxy behavior by default.
Changes:
- Adds collective GPUNetIO service setup and multi-QP device operations.
- Integrates the upstream GPUNetIO submodule and build/install configuration.
- Adds CPU-stub, P2P, and bandwidth tests plus usage documentation.
| File | Description |
|---|---|
.gitmodules |
Registers the GPUNetIO submodule. |
CMakeLists.txt |
Adds and validates the build option. |
docs/quickstart.md |
Documents optional backend setup. |
include/mscclpp/gpu_net_io_service.hpp |
Declares the host service API. |
include/mscclpp/internal/port_channel_gpunetio_device_impl.hpp |
Implements device RDMA operations. |
include/mscclpp/port_channel.hpp |
Adds GPUNetIO channel construction. |
include/mscclpp/port_channel_device.hpp |
Routes device operations by backend. |
include/mscclpp/port_channel_gpunetio_device.hpp |
Defines the device context API. |
src/CMakeLists.txt |
Includes the optional component. |
src/core/CMakeLists.txt |
Links GPUNetIO objects. |
src/core/include/ib.hpp |
Exposes internal verbs handles. |
src/core/port_channel.cc |
Constructs GPUNetIO channels. |
src/gpunetio/CMakeLists.txt |
Builds and installs the dependency. |
src/gpunetio/PROVENANCE.md |
Documents provenance and limitations. |
src/gpunetio/host/gpu_net_io_qp_table.hpp |
Builds sparse device QP tables. |
src/gpunetio/host/gpu_net_io_service.cpp |
Sets up QPs, memory, and device state. |
src/gpunetio/vendor |
Pins the upstream GPUNetIO dependency. |
test/mp_unit/CMakeLists.txt |
Enables GPUNetIO MP tests. |
test/mp_unit/port_channel_tests.cu |
Adds P2P and bandwidth coverage. |
test/unit/CMakeLists.txt |
Registers focused unit tests. |
test/unit/gpunetio_channel_test.cc |
Tests routing, QPs, and host API behavior. |
💡 Configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
Refactor flush logic to handle errors from tryFlush. Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
Construct GPUNetIO host channels through the service to validate peers against retained rank/world-size metadata and reject incomplete setup. Restrict QP creation to direct GPU doorbells, valid DBRs and GPU-resident CQs; remove CPU progress fallback and exchange QP failures before connection-info collectives. Add host peer-validation and direct-only policy regression coverage. Preserve the unmodified upstream submodule. ON/OFF builds, eight CPU test runs and repository lint pass; direct-GPU hardware qualification remains separate.
Replace the Git submodule with ON-only FetchContent pinned to the full upstream v4.0.1 SHA. Share dependency include paths through an interface target and retain offline overrides and installed headers. Remove unused DO_PRAGMA and NVCC_PRAGMA_UNROLL compatibility definitions. Add CUDA compile probes for clean and caller-defined macro environments. Validate ON/OFF builds, eight CPU tests, disconnected overrides, installed headers, and lint.
Expose explicit connection handles for existing QPs, independently registered local/remote memory, and owned semaphore counters exchanged per connection. Preserve per-channel memory IDs and use the bound QP for writes, signals, atomics and flushes while retaining resource lifetimes. Migrate P2P and multi-QP bandwidth tests to the public binding API and add two-channel isolation coverage. ON/OFF builds, eight CPU tests, installed-header checks and lint pass. GPU/NIC execution remains pending; uniform collective QP setup is retained.
Rename gin-prefixed device members, constructor parameters and helpers to gpuNetIo and update their test references. Remove NCCL-specific comparisons from public backend comments. No behavior or layout changes. Validated naming-only equivalence, ON/OFF builds, eight CPU tests and repository lint.
There was a problem hiding this comment.
Copilot review overview
🔵 Needs a closer look
The GPU/NIC transport remains hardware-unverified, and the review found documentation, licensing, and build-diagnostic issues.
Review effort: Balanced
Findings: 2
Open (5)
Resolved since last review (7)
Reject self and out-of-range peer ranks during construction Avoid enabling backend with broken pinned upstream implementation Release builds discard EBUSY instead of completing bounded flush Validate peer before indexing peer arrays Add required MIT license header to C++ test file Add required MIT license header to C++ header Add required MIT license header to public C++ header
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
There was a problem hiding this comment.
🟡 Changes recommended
Unresolved issues can silently disable Device-mode transfers and hang peer waits for installed-header consumers, while missing optional mlx5 headers can break otherwise supported IB builds.
5 open issues
Guard optional mlx5 include directories against NOTFOUND values · New Propagate GPUNETIO configuration to installed-header consumers · New Prevent mixed-mode connections from exceeding responder atomic depth · New Reject unregistered memory before returning device handles · New Allocate semaphore counters on the connection's local GPU · New
5 resolved since last review
Review effort: Balanced
GPUNetIO queue pairs are created on an IB device's protection domain, so the DOCA device and per-GPU handles opened on it now belong to that IbCtx (IbCtx::gpuNetIoDevice(), created on first use and released before the protection domain). Context no longer keeps a separate map of GPUNetIO contexts. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 95bbe000-7818-4c60-8184-773c93680b01
There was a problem hiding this comment.
🔵 Needs a closer look
The change breaks CMake generation when optional MLX5 headers are missing and allocates semaphore counters on the wrong GPU when the current device differs from the connection’s device, risking illegal memory access.
5 open issues
Propagate GPUNETIO configuration to installed-header consumers Guard optional mlx5 include directories against NOTFOUND values Allocate semaphore counters on the connection's local GPU Reject unregistered memory before returning device handles Prevent mixed-mode connections from exceeding responder atomic depth
Review effort: Balanced
Configure per-CTA WQE capacity through device-handle templates and accept runtime slot indices with doorbell fallback. Remove caller-owned shared-memory binding and update existing callers and documentation. Drop the local upstream CQ cleanup patch and track its error-handling issues upstream. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 55ef1b7e-0ccb-4b59-ad58-48a168608302
There was a problem hiding this comment.
🟡 Changes recommended
Unresolved critical defects include a double-free path, wrong-GPU memory allocation, silent communication no-ops for installed consumers, and a polling race that can consume the same signal twice.
9 open issues
Prevent inline writes from reading inaccessible host memory · New Atomically claim each inbound signal only once · New Propagate GPUNETIO feature configuration to installed consumers · New Reject memory handles without an address for the channel transport · New Propagate GPUNETIO configuration to installed-header consumers Guard optional mlx5 include directories against NOTFOUND values Allocate semaphore counters on the connection's local GPU Reject unregistered memory before returning device handles Prevent mixed-mode connections from exceeding responder atomic depth
Review effort: Balanced
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 55ef1b7e-0ccb-4b59-ad58-48a168608302
There was a problem hiding this comment.
🟡 Changes recommended
Unresolved issues can break builds without optional MLX5 headers and silently turn GPUNetIO device operations into no-ops for installed-library consumers.
10 open issues
Reject memory handles without an address for the channel transport Propagate GPUNETIO feature configuration to installed consumers Atomically claim each inbound signal only once Prevent inline writes from reading inaccessible host memory Propagate GPUNETIO configuration to installed-header consumers Guard optional mlx5 include directories against NOTFOUND values Compile the patched upstream file and document the workaround · New Allocate semaphore counters on the connection's local GPU Reject unregistered memory before returning device handles Prevent mixed-mode connections from exceeding responder atomic depth
Review effort: Balanced
…elDeviceHandle Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 55ef1b7e-0ccb-4b59-ad58-48a168608302
There was a problem hiding this comment.
🔵 Needs a closer look
Unresolved critical defects allocate semaphore state on the wrong GPU, risking illegal access or hangs, and break CUDA+IB configuration when optional MLX5 headers are absent.
10 open issues
Reject memory handles without an address for the channel transport Propagate GPUNETIO feature configuration to installed consumers Atomically claim each inbound signal only once Prevent inline writes from reading inaccessible host memory Propagate GPUNETIO configuration to installed-header consumers Guard optional mlx5 include directories against NOTFOUND values Compile the patched upstream file and document the workaround Allocate semaphore counters on the connection's local GPU Reject unregistered memory before returning device handles Prevent mixed-mode connections from exceeding responder atomic depth
Review effort: Balanced
…eHandle Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 55ef1b7e-0ccb-4b59-ad58-48a168608302
There was a problem hiding this comment.
🟡 Changes recommended
Unresolved critical issues can break CMake generation, allocate semaphore counters on the wrong GPU, and cause installed consumers to compile assertion-only GPUNetIO stubs.
11 open issues
Reject memory handles without an address for the channel transport Propagate GPUNETIO feature configuration to installed consumers Atomically claim each inbound signal only once Prevent inline writes from reading inaccessible host memory Propagate GPUNETIO configuration to installed-header consumers Guard optional mlx5 include directories against NOTFOUND values Compile the patched upstream file and document the workaround Allocate semaphore counters on the connection's local GPU Reject unregistered memory before returning device handles Prevent mixed-mode connections from exceeding responder atomic depth Correct the GPUNetIO usage example for the current API · New
Review effort: Balanced
…_IBVERBS Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 55ef1b7e-0ccb-4b59-ad58-48a168608302
There was a problem hiding this comment.
🟡 Changes recommended
Unresolved defects route downstream device kernels to disabled stubs, allow concurrent pollers to consume the same signal twice, and can double-free CQ buffers on initialization failure.
2 open issues
11 resolved since last review
Reject memory handles without an address for the channel transport Propagate GPUNETIO feature configuration to installed consumers Atomically claim each inbound signal only once Prevent inline writes from reading inaccessible host memory Propagate GPUNETIO configuration to installed-header consumers Guard optional mlx5 include directories against NOTFOUND values Compile the patched upstream file and document the workaround Allocate semaphore counters on the connection's local GPU Reject unregistered memory before returning device handles Prevent mixed-mode connections from exceeding responder atomic depth Correct the GPUNetIO usage example for the current API
Review effort: Balanced
| #include "atomic_device.hpp" | ||
| #endif | ||
|
|
||
| #if defined(MSCCLPP_DEVICE_CUDA) && defined(USE_IBVERBS) |
| MSCCLPP_DOCA_THROW(doca_verbs_qp_attr_set_ack_timeout(attr, 18)); | ||
| MSCCLPP_DOCA_THROW(doca_verbs_qp_attr_set_retry_cnt(attr, 7)); | ||
| MSCCLPP_DOCA_THROW(doca_verbs_qp_attr_set_rnr_retry(attr, 7)); | ||
| MSCCLPP_DOCA_THROW(doca_verbs_qp_attr_set_max_rd_atomic(attr, 1)); |
…hannel Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 55ef1b7e-0ccb-4b59-ad58-48a168608302



Summary
EndpointConfig::Ib::Mode::Device, backed by DOCA GPUNetIO / GDAKI) toPortChannel.FetchContent, pinned to commitc5d97fd52346763bcfeaff593dd1d84215156218. Known upstream CQ error-path issues are tracked in NVIDIA-DOCA/gpunetio#23.PortChannelBackend(BasePortChannel<Backend>,PortChannel<Backend>,BasePortChannelDeviceHandleT<Backend, SmemSlots>, andPortChannelDeviceHandleT<Backend, SmemSlots>withHostandDevicespecializations, withBasePortChannelandPortChanneldefaulting toPortChannelBackend::Host, plus non-templated aliasesBasePortChannelDeviceHandleandPortChannelDeviceHandle).PortChanneldirectly fromstd::shared_ptr<PortDevice2DeviceSemaphore>(orSemaphore) andRegisteredMemory(PortChannel channel(semaphore, dst, src)) withoutProxyService, and obtain the device handle viachannel.deviceHandle()(orchannel.deviceHandle<SmemSlots>()).putWithAccumulateposts a write and an atomic together (putWithSignaluses it on the semaphore).PortChannelDeviceHandle::Concurrency(Deviceby default,Block,Exclusive).Exclusiveoperations submit through BlueFlame up toblueFlameLimit(defaultkDefaultBlueFlameLimit = 512 KiB) using internal 64-byte shared-memory slots (SmemSlots, default 32 per CTA); larger transfers and the sharedBlock/Devicemodes ring the regular doorbell (submit_db).Build and Use
GPUNetIO is built automatically whenever
MSCCLPP_USE_CUDA=ON,MSCCLPP_USE_ROCM=OFF, andMSCCLPP_USE_IB=ON(the default), and skipped for ROCm or-DMSCCLPP_USE_IB=OFFbuilds:For an offline GPUNetIO checkout:
cmake -S . -B build -DFETCHCONTENT_SOURCE_DIR_GPUNETIO=/absolute/path/to/gpunetioUsage Example
In device code:
See
docs/guide/gpunetio.mdfor full details.Validation
Performance: GPUNetIO (
IbMode::Device) vs. IB CPU Proxy (IbMode::Host/IbMode::HostNoAtomic)Measured with
mp_unit_tests(PortChannelOneToOneTest) on 2x H100 GPUs over InfiniBand (kDefaultBlueFlameLimit = 512 KiB).One Thread per QP (
Concurrency::Exclusive, BlueFlame)Host)HostNoAtomic)Exclusive)HostputWithSignal+wait+flush)Host)HostNoAtomic)Exclusive)Shared QPs (
Concurrency::Block/Device, regular doorbell)The proxy runs the same kernels with the same number of threads per QP.
Host)Block)Device)In
BlockandDevicemodes, each posting thread updates the shared queue pair with GPU atomics and fences, and submits without BlueFlame. For the lowest latency, give each posting thread its own QP inExclusivemode.