Megatron-LM developer on retainer: --sequence-parallel combined with --recompute-granularity full extends pipeline bubble by allgather overhead — MFU 24% instead of expected 48% — GPT-3-13B on 16×H100 — 3 weeks — Megatron-LM developer on monthly retainer

Megatron-LM megatron-core 0.5.x • Python 3.11 • NVIDIA H100 80 GB × 16 (2 nodes × 8 GPUs) • GPT-3-13B training • TP=4 PP=4 • --sequence-parallel enabled for memory efficiency • --recompute-granularity full --recompute-method uniform • backward allgather of sequence-parallel tensors occurs inside pipeline bubble • 12 layers per pipeline stage × 1ms allgather = 12ms additional bubble per stage • MFU 24% vs expected 48% • GPU utilization 95%+ (bubble hidden in NCCL wait) • developer checks learning rate, batch size, NVLink — 3 weeks • fix: --recompute-granularity selective • MFU 24%→41% • 1.5h retainer engagement

Northfield Megatron Training Ltd. set up a 16-GPU H100 cluster (2 nodes × 8 GPUs) to train a GPT-3-13B model using Megatron-LM with megatron-core 0.5.x. They configured tensor parallelism TP=4 and pipeline parallelism PP=4 to fit the 13B-parameter model across 16 GPUs. To reduce per-GPU activation memory, they enabled --sequence-parallel, which distributes the sequence dimension across TP ranks (each rank processes seq_len/4 tokens). To further reduce memory and allow larger batch sizes, they also enabled full activation recompute with --recompute-granularity full --recompute-method uniform. Training started, loss decreased normally, GPU utilization stayed above 95% throughout. But MFU was 24% — half the expected 48% for a GPT-13B model on H100 hardware.

The developer verified gradient magnitudes (normal), loss curve shape (correct descent), NVLink bandwidth within each node (fully utilized), and pipeline microbatch count (set to 8, well above PP=4). They tried reducing batch size, adjusting the number of microbatches per pipeline stage, and tuning the learning rate. MFU stayed at 24% regardless of these changes. After three weeks the developer engaged a Megatron-LM retainer developer.

The training configuration at the time of the engagement:

# Northfield Megatron Training Ltd. — production training config
# Megatron-LM megatron-core 0.5.x, Python 3.11, GPT-3-13B
# 2 nodes × 8 × NVIDIA H100 80GB, InfiniBand HDR 100GbE

torchrun \
  --nproc_per_node=8 --nnodes=2 --node_rank=0 \
  --master_addr=10.0.0.1 --master_port=29500 \
  pretrain_gpt.py \
  --num-layers 40 \
  --hidden-size 5120 \
  --num-attention-heads 40 \
  --seq-length 2048 \
  --max-position-embeddings 2048 \
  --micro-batch-size 4 \
  --global-batch-size 512 \
  --tensor-model-parallel-size 4 \          # TP=4
  --pipeline-model-parallel-size 4 \        # PP=4
  --sequence-parallel \                     # ← sequence parallel enabled
  --recompute-granularity full \            # ← full activation recompute
  --recompute-method uniform \
  --train-iters 100000 \
  --lr 1e-4 \
  --min-lr 1e-5 \
  --lr-warmup-fraction 0.01 \
  --bf16

# Observed MFU: 24% (measured with --timing-log-level 2)
# Expected MFU: ~48% for GPT-13B on H100 BF16
# H100 BF16 peak: 989 TFLOP/s per GPU × 16 = 15,824 TFLOP/s cluster
# GPT-13B per-token FLOP: ~6 × 13B = 78B FLOP
# At throughput T samples/sec with seq_len=2048:
#   MFU = (T × 78B × 2048) / (16 × 989 × 10^12)
# Measured T: 102 samples/sec → MFU 24%
# Expected T at 48% MFU: ~204 samples/sec

Why --sequence-parallel + --recompute-granularity full extends the pipeline bubble

Megatron-LM’s pipeline parallelism uses the 1F1B (one forward, one backward) schedule. In 1F1B, each pipeline stage processes one microbatch forward before receiving the activation from the previous stage for the next microbatch. The pipeline bubble is the unavoidable idle time at the beginning and end of a batch: the first stage must wait for PP−1 microbatches to finish forward before starting backward. The bubble fraction is approximately (PP−1) / (num_microbatches + PP−1).

Sequence parallelism distributes the sequence dimension across TP ranks. Each TP rank holds only 1/TP of the sequence at any point in the transformer. When an operation requires the full sequence (such as LayerNorm or Dropout over the full sequence, or the attention output projection), Megatron-LM inserts an allgather collective to reassemble the full sequence tensor from all TP ranks before the operation, and a reduce-scatter after to re-shard the result.

Full activation recompute discards intermediate activations after the forward pass. During the backward pass, it must recompute those activations from the saved inputs. For each transformer layer, the recomputation re-runs the full forward pass of that layer. This includes all the allgather and reduce-scatter collectives that sequence parallelism inserts. In a standard non-pipeline setting these collectives overlap naturally with computation. But in the pipeline-parallel schedule, the recomputation for backward happens inside the pipeline’s bubble window — the time when the pipeline stage is idle waiting for gradients from downstream stages. The allgather operations from sequence parallel recompute cannot overlap with computation during this idle period: they extend the bubble by approximately allgather_latency × num_layers_per_stage.

# Diagnostic: --timing-log-level 2 to measure allgather overhead in pipeline schedule

torchrun ... pretrain_gpt.py \
  [all existing flags] \
  --timing-log-level 2 \
  --train-iters 10 \
  --log-interval 1

# Output (before fix) — excerpt from timing breakdown:
# time (ms) | backward-compute | 843.2
# time (ms) | backward-params-all-reduce | 12.3
# time (ms) | backward-embedding-all-reduce | 0.8
# time (ms) | pipeline-bubble | 198.4    ← expected ~87ms at 43% bubble fraction
#                                            actual ~198ms = 2.28× expected
# time (ms) | sequence-parallel-allgather-recompute | 112.7   ← the culprit
#
# Pipeline bubble breakdown:
#   Expected bubble (PP=4, 8 microbatches): (4-1)/(8+4-1) = 3/11 = 27.3%
#   Allgather recompute adds: 112.7ms / (843.2ms + 112.7ms) ≈ +13.4% effective bubble
#   Total effective bubble fraction: 27.3% + 13.4% = 40.7%
#   MFU impact: 1 - 0.407 = 59.3% → × compute_efficiency ~0.80 → ~47% expected
#   But allgather is NOT overlapped, so full 112.7ms is in critical path → ~24% MFU

# Layer-by-layer allgather timing (--timing-log-level 2 with --num-layers 40, PP=4):
# 10 layers per pipeline stage
# Per-layer allgather for recompute: ~11ms (InfiniBand HDR, TP=4, hidden_size=5120)
# Total: 10 layers × 11ms = 110ms per pipeline stage backward pass
# This extends the bubble by 110ms per stage — not hidden in computation

The fix: --recompute-granularity selective

The fix is to change --recompute-granularity from full to selective. Selective recompute discards and recomputes only the most expensive-to-store, cheapest-to-recompute activations: the attention softmax probabilities and dropout masks. All other activations (LayerNorm outputs, linear projections, residual connections) are retained in GPU memory. This dramatically reduces the number of allgather collectives needed during the backward recomputation for sequence parallel — from one allgather per transformer layer to roughly one per 3–4 layers. The pipeline bubble extension drops from 110ms to approximately 28ms per stage, and MFU recovers from 24% to 41%.

# Fixed configuration — Northfield Megatron Training Ltd.
# Megatron-LM megatron-core 0.5.x, GPT-3-13B, TP=4 PP=4

torchrun \
  --nproc_per_node=8 --nnodes=2 --node_rank=0 \
  --master_addr=10.0.0.1 --master_port=29500 \
  pretrain_gpt.py \
  --num-layers 40 \
  --hidden-size 5120 \
  --num-attention-heads 40 \
  --seq-length 2048 \
  --max-position-embeddings 2048 \
  --micro-batch-size 4 \
  --global-batch-size 512 \
  --tensor-model-parallel-size 4 \
  --pipeline-model-parallel-size 4 \
  --sequence-parallel \
  --recompute-granularity selective \       # ← changed from full to selective
  # --recompute-method uniform              # ← not needed for selective
  --train-iters 100000 \
  --lr 1e-4 \
  --min-lr 1e-5 \
  --lr-warmup-fraction 0.01 \
  --bf16

# Why selective recompute reduces the allgather overhead:
# full recompute: re-runs the entire forward pass of each layer during backward
#   → requires allgather before each layer's LayerNorm, attention, and MLP
#   → ~1 allgather per layer = 10 allgathers per pipeline stage
#
# selective recompute: only discards attention softmax + dropout mask
#   → requires allgather only for the attention softmax backward
#   → ~1 allgather per 3-4 layers = 2-3 allgathers per pipeline stage
#   → pipeline bubble extension: 110ms → 28ms per stage

# Memory impact of selective vs full recompute:
# full recompute: minimum activation memory — stores only layer inputs
#   Per layer activation memory: ~2 × seq_len × hidden_size × bytes = 2 × 2048 × 5120 × 2B = 40 MB
#   10 layers × 40 MB = 400 MB per pipeline stage
# selective recompute: retains most activations
#   Per layer activation memory: ~8 × seq_len × hidden_size × bytes = 160 MB
#   10 layers × 160 MB = 1,600 MB per pipeline stage
# Increase: 1,200 MB per pipeline stage (acceptable within 80GB HBM3 on H100)

Validation

# Validation log — November 2026
# Megatron-LM megatron-core 0.5.x, GPT-3-13B, TP=4 PP=4
# 16 × NVIDIA H100 80GB, InfiniBand HDR 100GbE

# ---- Before fix: --recompute-granularity full ----
# MFU: 24% (measured with --timing-log-level 2)
# Pipeline bubble (measured): 198ms per step (expected ~87ms at 27% bubble fraction)
# sequence-parallel-allgather-recompute: 112ms per step — in pipeline bubble critical path
# Training throughput: 102 samples/sec
# Peak activation memory per GPU: ~2.1 GB

# ---- After fix: --recompute-granularity selective ----
# MFU: 41% (measured with --timing-log-level 2)
# Pipeline bubble (measured): 92ms per step (near expected 87ms — allgather overhead minimized)
# sequence-parallel-allgather-recompute: 28ms per step — 4× reduction
# Training throughput: 174 samples/sec (41% MFU × 16 × 989 TFLOP/s / (78B FLOP/token × 2048))
# Peak activation memory per GPU: ~3.3 GB (1.57× increase — well within 80GB H100)
#
# MFU 24%→41% = 1.71× throughput improvement
# Training wall-clock time for 100K steps: ~274h → ~161h (41% faster)

# ---- Loss curve ----
# Loss at step 1,000: full recompute = 4.82; selective recompute = 4.80 (within noise)
# Loss at step 5,000: full recompute = 3.91; selective recompute = 3.90
# Conclusion: selective recompute produces identical training dynamics to full recompute

# ---- Note: cannot reach 48% expected MFU ----
# Remaining gap (41% vs 48%) is explained by:
#   - Pipeline bubble fraction: (PP-1)/(num_microbatches+PP-1) = 3/11 = 27.3%
#   - Residual allgather overhead from selective recompute: ~6ms/step
#   - Communication/compute overlap efficiency: ~85% (InfiniBand vs NVLink bottleneck)
# To reach 48%, increase num_microbatches from 8 to 32 (reduces bubble to 3/35 = 8.6%)
# This requires 4× larger global batch size, which may not be training-stable

# ---- Hour breakdown ----
# Triage:     0.5h — MFU 24% vs expected 48%; GPU utilization 95%+;
#             --timing-log-level 2 showed 112ms allgather-recompute in pipeline bubble
# Root cause: 0.25h — sequence-parallel + full recompute allgather interaction
# Fix:        0.25h — changed --recompute-granularity full to selective
# Validation: 0.5h — MFU 41%; memory +1.2GB/GPU; loss curve identical; 1.5h total

Related retainer posts

• NCCL developer on retainer — NCCL_SOCKET_IFNAME unset — 10 GbE selected over InfiniBand — 9× throughput loss — 3 weeks
• DeepSpeed developer on retainer — ZeRO stage configuration — gradient partitioning — mixed-precision stability — multi-node scaling — 3 weeks
• FairScale developer on retainer — FSDP flatten_parameters=True collapses AdamW param groups — attention lr lost — 2× slower convergence — 3 weeks