Perf: NCCL_BUFFSIZE=8M + IB timeout/retry/TOS in DeepSeek cycle env (AICAD T1aM4 evidence)

[Saolence]

Summary

Add the NCCL transport parameters that the AICAD DGX-Spark TP4 project proved out, and that are still missing from the DeepSeek cycle env templates: NCCL_BUFFSIZE=8388608 (8 MB pipe deepening), NCCL_IB_TIMEOUT=1000, NCCL_IB_RETRY_CNT=7, NCCL_IB_TOS=46. Pure environment — zero code change, one relaunch to A/B.

Evidence (AICAD, same topology)

Their T1aM4 step = Simple + MIN_CH4 + BUFFSIZE 8M took the 368 KB allreduce from 923 µs → 410 µs (-56%) before the later channel/tuner work. SparkRing already pins NCCL_MIN_NCHANNELS=4/MAX_NCHANNELS=4 (the AICAD end-state channel count), but the 8 MB buffer and the IB reliability/QoS knobs are absent from deepseek-v4-flash-0731*.env.example and the deployed rank envs.

AICAD production values (their config/production-nccl-env.md):

NCCL_BUFFSIZE=8388608
NCCL_IB_TIMEOUT=1000
NCCL_IB_RETRY_CNT=7
NCCL_IB_TOS=46

(Their later B1 work then found MAX_CH 16→4 helped large messages further; SparkRing already runs CH4, so this issue is only about the missing buffer/IB knobs on top of the existing CH4.)

Current state in sparkring

• scripts/config/deepseek-v4-flash-0731.env.example and -pair.env.example: no NCCL_BUFFSIZE, no NCCL_IB_TIMEOUT / RETRY_CNT / TOS.

Proposed change

Add to the DeepSeek env templates (and the deployed rank envs when A/B-ing):

NCCL_BUFFSIZE=8388608
NCCL_IB_TIMEOUT=1000
NCCL_IB_RETRY_CNT=7
NCCL_IB_TOS=46

A/B with the same nonce-fronted decode/prefill benchmark as the other proposals (c1/c4/c8 decode aggregate, 32K/131K prefill).

Reference implementation

• https://github.com/luxingcom/aicad-nccl-optimization — config/production-nccl-env.md; results in docs/reports/00-FINAL-REPORT-nccl-optimization-2026-08-16.md (T1aM4 -56% on the 368 KB allreduce; E2E 32K prefill +7.7–11.3% at that step).

Risks / notes

• Env-only; rollback is deleting the four lines and relaunching.
• TOS=46 sets an IP DSCP/QoS marking — confirm the fabric/switchless path doesn't drop marked packets (on a direct-cable ring there is no switch, so verify it is inert-or-beneficial rather than harmful).
• Keep NCCL_PROTO untouched (the per-size tuner proposal is separate).

Verification

• Allreduce latency at 4 KB–1 MB via nccl-tests (or E2E decode/prefill deltas) with the four vars vs baseline.
• Long-soak to confirm the IB timeout/retry values do not change failure behavior on the switchless ring.