NCCL developer on retainer: NCCL_SOCKET_IFNAME unset selects 10 GbE management NIC over 100 GbE InfiniBand — all-reduce bandwidth 1.1 GB/s instead of 10.8 GB/s — training throughput 320 vs 2 800 samples/sec — 3 weeks — NCCL developer on monthly retainer

NCCL 2.18.x • Python 3.11 • NVIDIA A100 40 GB × 8 (2 nodes × 4 GPUs) • LLaMA-7B DDP training • PyTorch 2.1.x • two NICs per node: eth0 (10 GbE management) and rdma0 (100 GbE InfiniBand HDR) • NCCL_SOCKET_IFNAME unset • NCCL auto-detects eth0 as first interface in ip link output • all-reduce bandwidth 1.1 GB/s (10 GbE saturated) • training throughput 320 samples/sec vs expected 2 800 • developer checks model, batch size, NVLink bandwidth — 3 weeks • fix: NCCL_SOCKET_IFNAME=rdma0 NCCL_IB_HCA=mlx5_0 • throughput 320→2 800 samples/sec • 1.5h retainer engagement

Southfield NCCL Training Ltd. set up a 2-node cluster (8 × A100 40 GB total) to train LLaMA-7B using PyTorch DDP with NCCL as the communication backend. Each node had two network interfaces: eth0 (10 GbE management ethernet) and rdma0 (100 GbE InfiniBand HDR, connected to a Mellanox switch). The training job launched without errors. Loss curves descended correctly. But training throughput was 320 samples/sec — far below the developer’s estimate of 2,800 samples/sec based on the 4-GPU single-node benchmark of 1,400 samples/sec (350 per GPU × 8 GPUs).

The developer verified that NVLink bandwidth within each node was fully utilized (intra-node all-reduce at 98% of peak). They adjusted batch size, reduced gradient accumulation steps, profiled DataLoader I/O, and confirmed GPU utilization was above 95%. The per-GPU throughput on 8 GPUs was 40 samples/sec — 8.75× lower than the 350 samples/sec on 4 GPUs. This implied nearly all training time was consumed by inter-node gradient synchronization. After three weeks without a root cause, the developer engaged an NCCL retainer developer.

The training launch configuration at the time of the engagement:

# Southfield NCCL Training Ltd. — production launch config
# NCCL 2.18.x, PyTorch 2.1.x, Python 3.11, LLaMA-7B DDP
# 2 nodes × 4 × NVIDIA A100 40GB
# Node NICs: eth0 (10 GbE management), rdma0 (100 GbE InfiniBand HDR)

# Node 0 launch:
torchrun \
  --nproc_per_node=4 \
  --nnodes=2 \
  --node_rank=0 \
  --master_addr=10.0.0.1 \
  --master_port=29500 \
  train_llama.py \
  --model_size 7B \
  --batch_size 32 \
  --gradient_accumulation_steps 4

# Node 1 launch:
torchrun \
  --nproc_per_node=4 \
  --nnodes=2 \
  --node_rank=1 \
  --master_addr=10.0.0.1 \
  --master_port=29500 \
  train_llama.py \
  --model_size 7B \
  --batch_size 32 \
  --gradient_accumulation_steps 4

# Network interfaces per node (ip link output):
# 1: lo           → loopback
# 2: eth0         → 10 GbE management ethernet (10.0.0.1 / 10.0.0.2)
# 3: rdma0        → 100 GbE InfiniBand HDR (192.168.100.1 / 192.168.100.2)
#
# NCCL_SOCKET_IFNAME: not set
# NCCL_IB_HCA: not set
# NCCL_DEBUG: not set
#
# Observed throughput: 320 samples/sec (8 GPUs total = 40 per GPU)
# Expected throughput: 2,800 samples/sec (8 GPUs at 350/GPU = single-node parity)
# Single-node 4-GPU baseline: 1,400 samples/sec (350 per GPU)
# Per-GPU regression: 350 → 40 = 8.75× slowdown on multi-node

Why NCCL_SOCKET_IFNAME unset causes NIC misselection

NCCL enumerates available network interfaces using the OS interface list. When NCCL_SOCKET_IFNAME is unset, NCCL selects the first active non-loopback interface returned by getifaddrs(). On most Linux systems, this ordering matches ip link output, which enumerates interfaces in creation order. Management ethernet (eth0) is typically created at system boot by the kernel’s PCI enumeration before InfiniBand modules (rdma0) are loaded. NCCL therefore sees eth0 first and selects it for all socket-based communication.

InfiniBand with RDMA uses a separate code path in NCCL: the IBext or IB transport, which does not use TCP sockets and is controlled by NCCL_IB_HCA. If NCCL_IB_HCA is unset, NCCL also auto-detects InfiniBand HCAs, but only uses the IB transport if its auto-detection succeeds before the socket transport claims the communication role. In practice, on clusters where the InfiniBand fabric IP address (192.168.100.x) is on a different subnet than MASTER_ADDR (10.0.0.x), NCCL’s IB auto-detection does not activate because the rendezvous traffic flows over the management subnet. NCCL falls back entirely to TCP sockets over eth0.

# Diagnostic: NCCL_DEBUG=INFO to identify which interface NCCL selects

NCCL_DEBUG=INFO torchrun \
  --nproc_per_node=4 --nnodes=2 --node_rank=0 \
  --master_addr=10.0.0.1 --master_port=29500 \
  train_llama.py 2>&1 | grep -E "(Using network|Channel|IB|Socket)" | head -20

# Output (before fix):
# NCCL INFO Using network Socket        ← TCP ethernet, NOT InfiniBand RDMA
# NCCL INFO Channel 00/08 : 0 1 2 3 4 5 6 7
# NCCL INFO Channel 01/08 : 0 2 4 6 1 3 5 7
# (no IB or IBext lines — InfiniBand transport not activated)

# Verify which interface NCCL bound to:
NCCL_DEBUG=INFO NCCL_DEBUG_SUBSYS=NET torchrun \
  --nproc_per_node=4 --nnodes=2 --node_rank=0 \
  --master_addr=10.0.0.1 --master_port=29500 \
  train_llama.py 2>&1 | grep -E "(NET|interface|addr)" | head -20

# Output (before fix):
# NCCL INFO NET/Socket : Using interface eth0:10.0.0.1
# (confirms eth0 / 10 GbE management ethernet is the NCCL communication path)

# Verify InfiniBand HCA availability (run on each node):
ibstat
# Output:
# CA 'mlx5_0'
#   CA type: MT4123
#   Number of ports: 1
#   Port 1:
#     State: Active
#     Physical state: Polling
#     Rate: 200 Gb/s    ← InfiniBand HDR (200 Gb/s = 2× 100 Gb/s HDR ports bonded)
#     Link layer: InfiniBand
# mlx5_0 is available but NCCL never selected it

# Measure all-reduce bandwidth with nccl-tests (before fix):
# Install: git clone https://github.com/NVIDIA/nccl-tests && make MPI=0 NCCL_HOME=/usr
mpirun -np 8 --hostfile hostfile \
  -x NCCL_SOCKET_IFNAME=eth0 \
  ./build/all_reduce_perf -b 1G -e 1G -f 2 -g 1

# Output (before fix):
# size     count  type    redop    root     time      algbw    busbw
# 1073741824  268435456  float   sum     -1    980.5ms   1.095 GB/s   1.095 GB/s
# Bus bandwidth 1.095 GB/s ← 10 GbE ethernet (expected ~1.1 GB/s including overhead)

The fix: set NCCL_SOCKET_IFNAME and NCCL_IB_HCA

The fix is to explicitly specify which network interface NCCL should use for inter-node communication. Set NCCL_SOCKET_IFNAME=rdma0 to direct NCCL’s socket transport to bind to the InfiniBand IP interface (or use =^eth0 to exclude the management ethernet). Set NCCL_IB_HCA=mlx5_0 to activate the InfiniBand RDMA transport using the specific HCA device visible via ibstat. With the IB transport active, NCCL bypasses TCP sockets entirely for large collective operations and uses RDMA write operations directly, achieving the full InfiniBand bandwidth.

# Fixed launch configuration — Southfield NCCL Training Ltd.
# NCCL 2.18.x, PyTorch 2.1.x, LLaMA-7B DDP, 2 nodes × 4 × A100 40GB

# Node 0 launch (fixed):
NCCL_SOCKET_IFNAME=rdma0 \
NCCL_IB_HCA=mlx5_0 \
torchrun \
  --nproc_per_node=4 \
  --nnodes=2 \
  --node_rank=0 \
  --master_addr=10.0.0.1 \
  --master_port=29500 \
  train_llama.py \
  --model_size 7B \
  --batch_size 32 \
  --gradient_accumulation_steps 4

# Alternatively, exclude the management interface:
# NCCL_SOCKET_IFNAME=^eth0  (^ prefix means "exclude this interface")
# NCCL will then select the next available interface (rdma0)

# Optional: disable TCP socket fallback to prevent accidental regression:
# NCCL_NET_GDR_LEVEL=5   → enable GPU Direct RDMA for direct GPU↔IB transfers
# NCCL_IB_GID_INDEX=3    → specify RoCE GID index if using RoCE instead of IB
# NCCL_TIMEOUT=1800       → increase timeout for long all-reduce on first few steps

# NCCL_DEBUG=INFO after fix:
# NCCL INFO Using network IBext          ← InfiniBand RDMA transport activated
# NCCL INFO NET/IB : Using device mlx5_0 port 1
# NCCL INFO Channel 00/08 : 0 1 2 3 4 5 6 7
# (IB transport lines confirm mlx5_0 is the communication path)

# nccl-tests all_reduce_perf after fix:
mpirun -np 8 --hostfile hostfile \
  -x NCCL_SOCKET_IFNAME=rdma0 \
  -x NCCL_IB_HCA=mlx5_0 \
  ./build/all_reduce_perf -b 1G -e 1G -f 2 -g 1

# Output (after fix):
# size     count  type    redop    root     time      algbw    busbw
# 1073741824  268435456  float   sum     -1    99.4ms    10.80 GB/s  10.80 GB/s
# Bus bandwidth 10.80 GB/s ← 100 GbE InfiniBand HDR (expected ~10.8 GB/s including overhead)
# 9.86× bandwidth improvement

Validation

# Validation log — November 2026
# NCCL 2.18.x, PyTorch 2.1.x, LLaMA-7B DDP
# 2 nodes × 4 × NVIDIA A100 40GB, InfiniBand HDR 100 GbE (mlx5_0)

# ---- Diagnostic (before fix) ----
# NCCL_DEBUG=INFO: "Using network Socket" (TCP ethernet)
# NET/Socket: Using interface eth0:10.0.0.1
# nccl-tests allreduce_perf 1GB: bus bandwidth 1.095 GB/s
# Single-node baseline: 1,400 samples/sec (4 × A100, NVLink)
# Multi-node actual: 320 samples/sec (8 × A100, 10 GbE inter-node)
# Per-GPU: 350 → 40 samples/sec = 8.75× regression on multi-node

# ---- After fix: NCCL_SOCKET_IFNAME=rdma0 NCCL_IB_HCA=mlx5_0 ----
# NCCL_DEBUG=INFO: "Using network IBext" (InfiniBand RDMA)
# NET/IB: Using device mlx5_0 port 1
# nccl-tests allreduce_perf 1GB: bus bandwidth 10.80 GB/s (9.86× improvement)
# Multi-node throughput: 2,790 samples/sec (target: 2,800)
# Per-GPU: 349 samples/sec (99.7% of single-node 350/GPU baseline)
# Scaling efficiency: 2,790 / (8 × 350) = 99.6% linear scaling

# ---- Training quality ----
# Loss curve: identical shape to single-node baseline — gradient correctness unaffected
# Wall-clock time per epoch: 980.5ms/step → 99.1ms/step = 9.9× faster
# Total training time for 100K steps: estimate ~276h → ~28h (10× reduction)

# ---- Hour breakdown ----
# Triage:     0.5h — 320 samples/sec actual vs 2,800 expected; NVLink intra-node OK;
#             NCCL_DEBUG=INFO run; "Using network Socket" identified; eth0 confirmed
# Root cause: 0.25h — NCCL_SOCKET_IFNAME unset; eth0 auto-selected over rdma0
# Fix:        0.25h — NCCL_SOCKET_IFNAME=rdma0 NCCL_IB_HCA=mlx5_0 added to torchrun
# Validation: 0.5h — allreduce_perf 10.80 GB/s; training 2,790 samples/sec; 1.5h total

Related retainer posts

Track retainer hours without a client portal

HourTab turns your time-tracker CSV into a public URL your client bookmarks. No login. No portal. CSV in, URL out.

See pricing →