MultimodalFlow
中文
← Back to Blog

Clustering Two NVIDIA DGX Spark Systems: 200GbE Looks Fast, but NCCL Tells the Real Story

DGX SparkNCCLRDMAvLLMmulti-node200GbERoCEdistributed inferenceLLMConnectX-7

I recently tested a two-node NVIDIA DGX Spark setup to understand whether the high-speed QSFP / ConnectX-7 interconnect is actually viable for serving large language models with vLLM. The short answer: the physical link is excellent, but the number that matters for LLM serving is not the one on the spec sheet.

Test Environment

Two NVIDIA DGX Spark systems, directly connected over the QSFP / ConnectX-7 high-speed port:

NodeIP
dgxse01192.168.10.1
dgxse02192.168.10.2

The RDMA interface configuration on both nodes:

SettingValue
Network interfaceenp1s0f0np0
RDMA devicerocep1s0f0
GID index3

Target workload: Qwen3.5-122B-A10B-FP8 served via vLLM with Ray as the distributed backend.

Step 1: Verify the Physical Link

Before testing RDMA or NCCL, I confirmed the QSFP link negotiated at full speed:

ethtool enp1s0f0np0 | grep -E "Speed|Link detected"

Result:

Speed: 200000Mb/s
Link detected: yes

200GbE confirmed. This only tells you the physical layer is healthy — it says nothing about what NCCL will actually achieve at the application level.

Step 2: Raw RDMA Bandwidth

Next, I measured the raw RDMA write bandwidth using ib_write_bw — the standard tool for characterizing InfiniBand / RoCE links.

On dgxse01 (server side):

ib_write_bw -d rocep1s0f0 -F --report_gbits -s 1048576 -q 8 -b

On dgxse02 (client side):

ib_write_bw -d rocep1s0f0 -F --report_gbits -s 1048576 -q 8 -b 192.168.10.1

Result:

BW average: ~197 Gb/s

Converting to GB/s: 197 / 8 ≈ 24.6 GB/s

At the RDMA layer, the Spark-to-Spark connection is healthy and running very close to the theoretical 200GbE ceiling.

Step 3: NCCL All-Reduce

Raw RDMA bandwidth is necessary but not sufficient for LLM serving. Frameworks like vLLM use NCCL for GPU-to-GPU collective communication — and NCCL collective bandwidth is a different number from point-to-point RDMA bandwidth.

I used nccl-tests to measure all_reduce_perf across both nodes:

cd /opt/nccl-tests
mpirun --allow-run-as-root -np 2 \
  -H 192.168.10.1:1,192.168.10.2:1 \
  --bind-to none \
  --map-by slot \
  --mca oob_tcp_if_include enp1s0f0np0 \
  --mca btl_tcp_if_include enp1s0f0np0 \
  -x NCCL_SOCKET_IFNAME=enp1s0f0np0 \
  -x NCCL_IB_DISABLE=0 \
  -x NCCL_IB_HCA=rocep1s0f0 \
  -x NCCL_IB_GID_INDEX=3 \
  -x NCCL_IB_TC=106 \
  -x NCCL_IB_PCI_RELAXED_ORDERING=1 \
  /opt/nccl-tests/build/all_reduce_perf -b 8M -e 4G -f 2 -g 1

Result:

Avg bus bandwidth: ~10.2 GB/s

That is roughly 40% of the raw RDMA bandwidth. The NCCL logs showed it was using the correct path:

NCCL INFO NET/IB
GID 3
devName=rocep1s0f0

NCCL was not falling back to TCP sockets — it was correctly using NET/IB. The lower bandwidth is not a misconfiguration.

Step 4: NCCL Send/Recv

Pipeline parallelism in vLLM depends more on point-to-point send/recv operations than on all-reduce. I tested that separately:

cd /opt/nccl-tests
mpirun --allow-run-as-root -np 2 \
  -H 192.168.10.1:1,192.168.10.2:1 \
  --bind-to none \
  --map-by slot \
  --mca oob_tcp_if_include enp1s0f0np0 \
  --mca btl_tcp_if_include enp1s0f0np0 \
  -x NCCL_SOCKET_IFNAME=enp1s0f0np0 \
  -x NCCL_IB_DISABLE=0 \
  -x NCCL_IB_HCA=rocep1s0f0 \
  -x NCCL_IB_GID_INDEX=3 \
  -x NCCL_IB_TC=106 \
  -x NCCL_IB_PCI_RELAXED_ORDERING=1 \
  /opt/nccl-tests/build/sendrecv_perf -b 8M -e 4G -f 2 -g 1

Result:

sendrecv_perf: ~9 GB/s

Point-to-point performance was not meaningfully faster than all-reduce. PP=2 is not guaranteed to outperform TP=2 on this interconnect.

Step 5: Understanding GDR 0

The most revealing line in the NCCL log was:

Connected all rings, use ring PXN 0 GDR 0

GDR 0 means NCCL is using the RDMA network path, but without GPUDirect RDMA. With GPUDirect disabled, GPU tensor data must travel through system memory before reaching the NIC — adding a PCIe copy step that the raw ib_write_bw test completely bypasses.

That is the root cause of the gap:

LayerBandwidth
ib_write_bw (raw RDMA)~24.6 GB/s
NCCL all_reduce_perf~10.2 GB/s
NCCL sendrecv_perf~9 GB/s

ib_write_bw measures host-to-host RDMA at full link speed. NCCL moves GPU tensors — a fundamentally different path when GPUDirect is not active.

Step 6: vLLM Deployment Strategy

Ray detected both nodes correctly:

Active nodes: 2
Total GPU:    2

For splitting Qwen3.5-122B-A10B-FP8 across two Spark systems, two strategies are available:

Option A: Tensor Parallelism (TP=2)

tensor-parallel-size: 2
pipeline-parallel-size: 1

Each layer is split across both systems. Every forward pass requires cross-node all-reduce operations. With ~10.2 GB/s all-reduce bandwidth, TP=2 can work, but the communication cost per layer is non-trivial for a 122B model.

Option B: Pipeline Parallelism (PP=2)

tensor-parallel-size: 1
pipeline-parallel-size: 2

Each system owns a contiguous slice of layers. Cross-node communication happens only at pipeline stage boundaries. Initially I expected PP=2 to win clearly, but since sendrecv_perf measured ~9 GB/s — almost identical to all-reduce — the advantage is not as large as expected on this interconnect.

The honest answer: benchmark both with your actual model, same prompts, same context length, same output length, and let the real token/s numbers decide.

Baseline Summary

LayerMetricResult
PhysicalLink speed200GbE
PhysicalLink detectedYes
RDMAib_write_bw~197 Gb/s (~24.6 GB/s)
NCCLVersion2.28.9
NCCLTransportNET/IB (rocep1s0f0, GID 3)
NCCLGPUDirectGDR 0 (disabled)
NCCLall_reduce_perf~10.2 GB/s
NCCLsendrecv_perf~9 GB/s
vLLMRay nodes2 nodes, 2 GPUs

Conclusion

The two-node DGX Spark cluster connected cleanly over the 200GbE / ConnectX-7 link. Raw RDMA bandwidth is excellent — nearly the full theoretical 200GbE ceiling. The network is not broken.

But the key lesson is simple:

200GbE does not automatically mean 25 GB/s for NCCL.

For large model serving, you are working across four distinct performance layers — physical link speed, raw RDMA bandwidth, NCCL collective bandwidth, and real vLLM token/s — and they can differ by 2–3× between adjacent layers.

In this setup, the effective GPU communication bandwidth available to vLLM is around 9–10 GB/s, not 25 GB/s. That is your real design constraint. For a 122B FP8 model across two Spark systems, test both PP=2 and TP=2 against your actual serving workload before committing to a parallelism strategy.

Measured June 19, 2026. NCCL 2.28.9, vLLM with Ray. Model: Qwen3.5-122B-A10B-FP8.