Report a dead peer in JACCL instead of spinning forever; name the lin…

[Sofille65]

Problem

When a rank of a JACCL group dies, the other ranks never find out. A UC queue pair on Apple's Thunderbolt RDMA stack produces no error completion and no event for a dead peer, and RC queue pairs are not available (ibv_create_qp fails with errno 102), so every completion wait loop in mesh_impl.h / ring_impl.h spins at 100 % CPU indefinitely. This is #4278.

Separately, the init errors hide their cause. On a device whose Thunderbolt port is PORT_DOWN, ibv_open_device succeeds but ibv_alloc_pd returns NULL, reported as a bare "Couldn't allocate protection domain". A peer whose IPv4-mapped GID is not reachable on this link fails RTR with errno 60, reported without device or address. And when one rank fails during init, the coordinator closes the side channel and every other rank dies with "Recv failed with errno=2": a stale errno from ::recv returning 0, which reads like an RDMA failure. (#3467 is the same family: a wrong GID / sgid_index gives RTR errno 22.)

Change

Peer liveness from the side channel. The TCP side channel lives as long as the group. TCPAllGather::fds() exposes its socket fds and Config::get_side_channel() passes them to SideChannel::liveness_fds(), which MeshGroup / RingGroup hand to MeshImpl / RingImpl. Every completion wait loop (6 in mesh, 4 in ring) owns a ProgressGuard and calls tick(n > 0) after ibv_poll_cq. While nothing completes, at most every 250 ms the guard poll()s the fds with a zero timeout and does a one-byte MSG_PEEK on POLLIN; POLLHUP/POLLERR/POLLNVAL or a zero-byte peek means the peer process is gone → std::runtime_error("[jaccl] peer is gone: side channel to rank N closed while waiting in <op> …"). It never reads, writes or locks, so it is safe from the ring wire threads. Rank 0 watches every peer; other ranks watch rank 0, whose throw closes its sockets, so a non-zero rank's death reaches everyone in two hops. When JACCL is initialised with a user-supplied all-gather (no TCP coordinator) the fd list is empty and only the timeout below applies.

Progress timeout backstop. JACCL_PROGRESS_TIMEOUT_S / MLX_JACCL_PROGRESS_TIMEOUT_S (default 600 s, 0 disables): no completion for that long → runtime_error("[jaccl] no progress in <op> for N s …"). Covers what the side channel cannot see: a lost UC frame, or a peer that is alive but wedged. The default is deliberately longer than any legitimate wait.

Diagnostics. alloc_pd failure names the device and its port state; RTR failure names the device, port state and the peer's IPv4 from the GID and decodes errno 60/22; side-channel EOF says "peer closed the side channel (another rank failed during init or exited)".

Happy path cost: tick(true) is two stores; the slow path runs after 256 empty polls and then at most every 250 ms. The stock 3-argument SideChannel constructor is kept as an overload, so the exported symbols of libjaccl are a superset of before (checked with nm -gU against the 0.32.2 wheel).

Testing

There is no way to exercise JACCL without Macs on Thunderbolt 5 with RDMA enabled, so this cannot be an automated test in python/tests. It was tested on five M3 Ultras in a full Thunderbolt 5 mesh (macOS 26.6.1, mlx 0.32.2 wheel with the rebuilt libjaccl.dylib dropped in on every node). First the isolated scenarios on a 2-node group:

Scenario	Stock	This PR
mesh, 50–200 × all_sum 16 MB, clean	OK	OK, same latency (1.9 ms)
mesh, rank 1 _exit() at iter 20	rank 0 spins forever	rank 0 raises "peer is gone … all_reduce" after 0.25 s
mesh, rank 0 (coordinator) _exit()	rank 1 spins forever	rank 1 raises "… rank 0 (coordinator) closed" after 0.25 s
ring (MLX_JACCL_RING=1), rank 1 _exit()	spins	raises "… reduce_scatter_wire" after 0.25 s
mesh, SIGKILL of rank 1 mid-run	spins	raises after 0.25 s
reconnect after each of the above	—	new group, 50–200 iters OK

Still on the 2-node group but inside a production orchestrator: a pipeline-parallel generation with a rank SIGKILLed mid-stream ends with the exception on the survivor in under a second, three times in a row, and the group re-forms afterwards.

Then on the 5-node group (five M3 Ultras, full Thunderbolt 5 mesh, 20 RDMA edges) serving GLM-5.3 (744B, 8-bit, pipeline parallel over JACCL):

• one rank crashed at the first token of a request (a stale model patch on that node, unrelated to JACCL); the four other ranks raised [jaccl] peer is gone: side channel to rank 4 closed while waiting in all_gather (rank 0) and … to rank 0 (coordinator) closed … (ranks 1–3) within the same second, the orchestrator purged and re-formed the group with no operator action, twice;
• the same 5-node group then ran a reasoning-heavy benchmark for hours with zero JACCL events and the same per-token latency as the stock library.

Facts behind the diagnostics, measured with a libibverbs-only program that dlopens librdma.dylib exactly like JACCL: 10 queue pairs per device (errno 16 on the 11th, shared across processes), ≥ 512 MRs, no leak of QPs/MRs across clean exit, _exit, or SIGKILL; alloc_pd NULL on a PORT_DOWN device; RTR errno 60 for an unreachable peer GID; RTR errno 22 for a zero GID. The probe sources are in https://github.com/Odyssai-eu/OdyssAI-X/tree/main/scripts/jaccl (rdma_probe.c, rdma_pair.c, smoke_jaccl.py).