Skip to content

tp2: render only the selected HCA functions into B12X_ROCE_HCA - #270

Open
SecureBot wants to merge 1 commit into
FujitsuPolycom:mainfrom
SecureBot:fix/tp2-single-dac-roce-hca
Open

tp2: render only the selected HCA functions into B12X_ROCE_HCA#270
SecureBot wants to merge 1 commit into
FujitsuPolycom:mainfrom
SecureBot:fix/tp2-single-dac-roce-hca

Conversation

@SecureBot

Copy link
Copy Markdown

What and why

transport_environment() in the TP2 launcher renders the full four-device hca_inventory into B12X_ROCE_HCA and narrows the selection only in B12X_ROCE_PEER_HCA_MAP / NCCL_IB_HCA. The RoCE proxy opens every device in B12X_ROCE_HCA and requires each port to be ACTIVE, so on a host with one physical DAC in cage p0 the two uncabled f1 functions abort RoCEnante setup and the engine silently falls back to PyNCCL:

WARNING [b12x_roce_all_reduce.py:102] RoCEnante initialization failed: ... RDMA device rocep1s0f1 port 1 is not active
INFO [cuda_communicator.py:314] Using ['PYNCCL'] all-reduce backends ... for group 'tp:0'

This change renders only the selected functions (rocep1s0f0,roceP2p1s0f0) and a peer map addressing that list (0/1). NCCL_IB_HCA, path count, and the profile/manifest hashes are unchanged; profile.json is not touched. With this change RoCEnante initializes as the first all-reduce backend on a single-cable install, matching the README's "both PCI functions of one physical DAC" description.

Fixes #268.

Validation

  • python -m pytest runtime/profiles/glm53-flash-spark-tp2 -q — 56 passed (two assertions in test_rank_plan_maps_both_pci_functions_of_one_cage and the rejected site-file peer-map example updated to the new rendering).
  • python -m pytest runtime/sparkring/jovian-r33/profiles -q — 33 passed.
  • Live: two DGX Spark (GB10), one QSFP112 DAC between port 0 of each, enp1s0f1np1 NO-CARRIER on both, R33 image 3c7779ad71dd… with --r33-sparkcache. Before: warning above on both ranks, ['PYNCCL']. After: no RoCEnante warning; B12X_ROCENANTE selected first for tp:0 and ep:0 on both ranks. FILL IN: any throughput comparison you have (configuration, prompt sizes, units, and that it's a single-site measurement).

I did not re-run the profile's full qualification record; this changes the rendered transport environment, so the maintainers may want to re-qualify or take the alternative of relaxing the ACTIVE requirement in _roce_proxy.c for unmapped devices instead.

Additional context

The tp2-single-dac-source-20260908 record notes both physical cables remained connected during the reference trial, so the single-cable case appears not to have been exercised before. The README transport row is updated to say the other cage may be uncabled.

  • I removed credentials and private site identifiers from the change.
  • I identified copied or adapted work and updated third-party notices when required. (No copied work.)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

TP2 profile: RoCEnante fails with a single physical DAC because B12X_ROCE_HCA lists all four functions

1 participant