Skip to content

tp2: render only the selected HCA functions into B12X_ROCE_HCA #270

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
FujitsuPolycom merged 3 commits into from
Sep 13, 2026
Merged

tp2: render only the selected HCA functions into B12X_ROCE_HCA #270

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
27d6b0d
tp2: render only the selected HCA functions into B12X_ROCE_HCA
SecureBot Sep 11, 2026
6db2bfc
Document peer-map positions in the selected TP2 device list
FujitsuPolycom Sep 13, 2026
53c8ade
Merge branch 'main' into fix/tp2-single-dac-roce-hca
FujitsuPolycom Sep 13, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 3 additions & 2 deletions runtime/profiles/glm53-flash-spark-tp2/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -76,7 +76,7 @@ records the common-source requirements and reference-trial limits.
| Scheduler | Eight sequences, 8,192 batched tokens, prefill interval 8 |
| Prefill | Token-sharded mHC and recurrent-checkpoint coalescing; sequential KDA projection |
| Graphs | `FULL_AND_PIECEWISE`, mode 0; `[1,2,4,8,12,16,20,24,28,32]` |
| Transport | HCA indices 0/2; two RoCEnante paths; 16 MiB all-reduce and all-gather input-shard limits |
| Transport | HCA indices 0/2 (`rocep1s0f0`, `roceP2p1s0f0`); only these two functions are rendered into `B12X_ROCE_HCA`, so the other cage may be uncabled; two RoCEnante paths; 16 MiB all-reduce and all-gather input-shard limits |
| NCCL | Verified 2.30.7; eight channels; `=rocep1s0f0,roceP2p1s0f0` |
| Multimodal / SparkCache | Four images, zero videos; SparkCache disabled |
| Lifecycle | Active 2 GiB host-memory guard; manual create/start; Docker restart `no` |
Expand Down Expand Up @@ -144,7 +144,8 @@ transport, memory-guard, and cache settings come from the profile.

Confirm this hardware inventory maps to the intended physical cage:
`rocep1s0f0,rocep1s0f1,roceP2p1s0f0,roceP2p1s0f1`. The reciprocal peer maps
are rank 0 `1=0/2` and rank 1 `0=0/2`; both selected functions belong to p0.
use positions in the rendered two-device list: rank 0 `1=0/1` and rank 1
`0=0/1`. These still select inventory functions 0 and 2, both belonging to p0.
Install the host memory-guard service and apply
[memory-guard.conf](memory-guard.conf) as its systemd drop-in. Both `create`
and `start` require the active guard's effective 2 GiB floor. The launcher
Expand Down
9 changes: 7 additions & 2 deletions runtime/profiles/glm53-flash-spark-tp2/launch.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -35,10 +35,15 @@ def transport_environment(rank: int) -> dict[str, str]:
raise ValueError("The profile's transport manifest has changed")
indices = transport["local_hca_indices"]
inventory = transport["hca_inventory"]
# Expose only the selected functions. The RoCE proxy opens every device in
# B12X_ROCE_HCA and requires each to be ACTIVE, so listing the full inventory
# aborts RoCEnante on a host with one physical DAC (the f1 functions have no
# carrier). Peer-map indices are positions in the rendered list.
selected = [inventory[index] for index in indices]
return {
"B12X_ROCE_HCA": ",".join(inventory),
"B12X_ROCE_HCA": ",".join(selected),
"B12X_ROCE_PAIR_PATHS": str(transport["path_count"]),
"B12X_ROCE_PEER_HCA_MAP": f"{1 - rank}=" + "/".join(map(str, indices)),
"B12X_ROCE_PEER_HCA_MAP": f"{1 - rank}=" + "/".join(str(i) for i in range(len(selected))),
"NCCL_IB_HCA": "=" + ",".join(inventory[index] for index in indices),
"NCCL_IB_MERGE_NICS": "0",
"NCCL_CROSS_NIC": "1",
Expand Down
9 changes: 6 additions & 3 deletions runtime/profiles/glm53-flash-spark-tp2/test_glm53_spark_tp2_profile.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -168,8 +168,11 @@ def value(flag):
def test_rank_plan_maps_both_pci_functions_of_one_cage(inputs, rank):
value = plan(inputs, rank)
env = value["environment"]
assert env["B12X_ROCE_HCA"] == "rocep1s0f0,rocep1s0f1,roceP2p1s0f0,roceP2p1s0f1"
assert env["B12X_ROCE_PEER_HCA_MAP"] == f"{1 - rank}=0/2"
# Only the selected functions are exposed to the RoCE proxy: it requires every
# listed device to be ACTIVE, and the f1 functions carry no cable on a
# single-DAC install. Peer-map indices address the rendered list.
assert env["B12X_ROCE_HCA"] == "rocep1s0f0,roceP2p1s0f0"
assert env["B12X_ROCE_PEER_HCA_MAP"] == f"{1 - rank}=0/1"
assert env["NCCL_IB_HCA"] == "=rocep1s0f0,roceP2p1s0f0"
assert env["B12X_ROCE_PAIR_PATHS"] == "2"
assert env["NCCL_MIN_NCHANNELS"] == env["NCCL_MAX_NCHANNELS"] == "8"
Expand Down Expand Up @@ -243,7 +246,7 @@ def test_tp2_clears_inherited_mesh_sitecustomize(inputs, tmp_path):
@pytest.mark.parametrize(
"assignment",
[
"B12X_ROCE_PEER_HCA_MAP=1=2/1",
"B12X_ROCE_PEER_HCA_MAP=1=1/0",
"VLLM_PLUGINS=sparkcache",
"SPARK_CACHE_ENABLED=1",
"VLLM_HOST_IP=duplicate",
Expand Down
Loading