tp2: render only the selected HCA functions into B12X_ROCE_HCA - #270
Open
SecureBot wants to merge 1 commit into
Open
tp2: render only the selected HCA functions into B12X_ROCE_HCA#270SecureBot wants to merge 1 commit into
SecureBot wants to merge 1 commit into
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What and why
transport_environment()in the TP2 launcher renders the full four-devicehca_inventoryintoB12X_ROCE_HCAand narrows the selection only inB12X_ROCE_PEER_HCA_MAP/NCCL_IB_HCA. The RoCE proxy opens every device inB12X_ROCE_HCAand requires each port to be ACTIVE, so on a host with one physical DAC in cage p0 the two uncabled f1 functions abort RoCEnante setup and the engine silently falls back to PyNCCL:This change renders only the selected functions (
rocep1s0f0,roceP2p1s0f0) and a peer map addressing that list (0/1).NCCL_IB_HCA, path count, and the profile/manifest hashes are unchanged;profile.jsonis not touched. With this change RoCEnante initializes as the first all-reduce backend on a single-cable install, matching the README's "both PCI functions of one physical DAC" description.Fixes #268.
Validation
python -m pytest runtime/profiles/glm53-flash-spark-tp2 -q— 56 passed (two assertions intest_rank_plan_maps_both_pci_functions_of_one_cageand the rejected site-file peer-map example updated to the new rendering).python -m pytest runtime/sparkring/jovian-r33/profiles -q— 33 passed.enp1s0f1np1NO-CARRIER on both, R33 image3c7779ad71dd…with--r33-sparkcache. Before: warning above on both ranks,['PYNCCL']. After: no RoCEnante warning;B12X_ROCENANTEselected first fortp:0andep:0on both ranks. FILL IN: any throughput comparison you have (configuration, prompt sizes, units, and that it's a single-site measurement).I did not re-run the profile's full qualification record; this changes the rendered transport environment, so the maintainers may want to re-qualify or take the alternative of relaxing the ACTIVE requirement in
_roce_proxy.cfor unmapped devices instead.Additional context
The
tp2-single-dac-source-20260908record notes both physical cables remained connected during the reference trial, so the single-cable case appears not to have been exercised before. The README transport row is updated to say the other cage may be uncabled.