ESPnet developer on retainer: batch_type=numel with mixed-length utterances (5 s–8 min) produces 964-utterance batches (19 000 tokens) for short clips and 53-utterance batches (6 000 tokens) for long clips — summed CTC+attention loss varies 3× between consecutive batches — training instability — val_WER 0.19 vs expected 0.11 — 3 weeks — ESPnet developer on monthly retainer

ESPnet 202301.x • Python 3.11 • PyTorch 2.1.x • Transformer-based ASR on 200-hour customer service audio • utterance durations 5 s to 8 min • batch_type=numel • batch_bins=4000000 • short batches: ~964 utterances, ~19 000 output tokens • long batches: ~53 utterances, ~6 000 output tokens • CTC + attention loss summed over all tokens • gradient magnitude 3× variance across batches • training instability • val_WER=0.19 (expected 0.11) • developer checks learning rate, batch_bins (2M, 4M, 8M), model depth for 3 weeks • fix: normalize loss by token count (total_loss / batch_tokens) • val_WER 0.19→0.11 • 1.5h retainer engagement

Westfield ESPnet ASR Ltd. was training a Transformer-based ASR model (12-layer encoder, 6-layer attention decoder, joint CTC + attention objective) using ESPnet2 on a 200-hour customer service call corpus. The corpus contained recordings ranging from 5-second voicemail fragments to 8-minute full call recordings, with a median utterance duration of 22 seconds. The developer configured batch_type=numel with batch_bins=4000000 in the ESPnet2 recipe YAML, following the ESPnet tutorial for large-corpus training. After 50 epochs of training on a single A100 GPU, validation WER plateaued at 0.19 against an expected 0.11 based on comparable published ESPnet results on similar-length clean speech corpora. The training loss curve showed persistent oscillations rather than smooth convergence. After three weeks of adjusting the learning rate scheduler, trying batch_bins values of 2M, 4M, and 8M, and increasing model depth from 12 to 18 encoder layers, the team engaged an ESPnet retainer developer. The root cause was batch_type=numel producing batches with dramatically different output token counts for short vs. long utterances, causing the summed CTC and attention loss to vary 3× in magnitude between consecutive gradient steps without any change in per-token learning signal.

The ESPnet2 training configuration at the time of the engagement:

# Westfield ESPnet ASR Ltd. — 200-hour customer service ASR
# ESPnet 202301.x, Python 3.11, PyTorch 2.1.x
# asr_config.yaml (excerpt)

# ---- Encoder / decoder ----
encoder: conformer
encoder_conf:
    output_size: 256
    num_blocks: 12
    input_layer: conv2d
decoder: transformer
decoder_conf:
    num_blocks: 6
    attention_heads: 4

# ---- Objectives ----
model_conf:
    ctc_weight: 0.3
    # CTC + attention loss = ctc_weight * ctc_loss + (1 - ctc_weight) * att_loss
    # Both ctc_loss and att_loss are SUMMED over all tokens in the batch.
    # ← Problem: summed loss magnitude = per-token-loss × num_tokens_in_batch.
    # Batch token count varies dramatically with batch_type=numel on mixed-length corpus.

# ---- Batch strategy ----
batch_type: numel
# numel fills each batch until:
#   total_frames × feat_dim × batch_count ≈ batch_bins
#
# For 83-dim Fbank features (n_mels=80 + 3 pitch):
#
#   Short utterances (5s = 50 frames at 10ms frame shift):
#     batch capacity = batch_bins / (50 frames × 83 features) ≈ 964 utterances per batch
#     avg tokens per 5-second utterance ≈ 20 (customer service: short phrases)
#     total batch tokens ≈ 964 × 20 ≈ 19,280 tokens
#     summed loss magnitude ≈ 19,280 × 1.0 nats ≈ 19.3 nats
#
#   Long utterances (90s = 900 frames at 10ms frame shift):
#     batch capacity = batch_bins / (900 frames × 83 features) ≈ 53 utterances per batch
#     avg tokens per 90-second utterance ≈ 115 (customer service: multi-turn dialogue)
#     total batch tokens ≈ 53 × 115 ≈ 6,095 tokens
#     summed loss magnitude ≈ 6,095 × 1.0 nats ≈ 6.1 nats
#
#   Loss ratio between extreme batches:
#     19.3 / 6.1 ≈ 3.2×
#
#   gradient_norm(short_batch) ≈ 3.2 × gradient_norm(long_batch)
#   With fixed lr=1e-4: optimizer overshoots after high-loss short batches,
#   undershoots after low-loss long batches → oscillating convergence.
batch_bins: 4000000

# ---- Optimizer ----
optim: adam
optim_conf:
    lr: 0.001
    weight_decay: 1.0e-06
scheduler: warmuplr
scheduler_conf:
    warmup_steps: 25000

# ---- Input features ----
frontend: default
frontend_conf:
    n_fft: 512
    win_length: 400
    hop_length: 160
n_mels: 80
# ← Total feature dimension: 80 mel bins + 3 pitch features = 83 dims
# Used in batch_bins capacity calculation above.

# ---- Results ----
# val_WER after 50 epochs: 0.19 (expected 0.11)
# Training loss curve: oscillates ±4.0 nats between consecutive epochs
# Gradient norm: coefficient of variation = 0.62 (high variance)

Why batch_type=numel creates 3× loss magnitude variance on mixed-length corpora

ESPnet2’s batch_type=numel strategy targets a fixed total number of acoustic feature elements per batch by computing batch_bins / (utterance_frames × feature_dim) as the number of utterances to pack. For a 200-hour customer service corpus, the utterance duration distribution spans from 5-second voicemail fragments (50 frames at 10 ms frame shift) to 8-minute full call recordings (4,800 frames). With batch_bins=4,000,000 and 83-dimensional Fbank features, a batch composed entirely of 5-second utterances contains 4,000,000 / (50 × 83) ≈ 964 utterances; a batch of 90-second utterances contains only 4,000,000 / (900 × 83) ≈ 53.

ESPnet2’s default loss computation sums the CTC cross-entropy and the attention decoder cross-entropy over all tokens in the batch before multiplying by the respective weights. The 964-utterance short batch contains approximately 19,280 output tokens (each 5-second customer service phrase averages 20 word-piece tokens), producing a total loss of roughly 19.3 nats at an initial per-token cross-entropy of ~1.0. The 53-utterance long batch contains approximately 6,095 tokens (each 90-second dialogue averages 115 tokens), producing a total loss of roughly 6.1 nats. Because the ESPnet2 Adam optimizer sees consecutive gradient steps with 3.2× difference in loss magnitude — without any difference in per-token learning signal — the update magnitude oscillates: the optimizer takes a large step after a short-utterance batch (over-correcting) and a small step after a long-utterance batch (under-correcting). The characteristic signature is a training loss curve that oscillates by ±3–4 nats between consecutive epochs, and a validation WER that plateaus well above the expected optimum because the model never converges to a stable parameter region.

# Diagnostic: audit batch token count distribution
# ESPnet 202301.x, Python 3.11

import espnet2
from espnet2.train.dataset import ESPnetDataset
from espnet2.samplers.numel_batch_sampler import NumelBatchSampler
import numpy as np

# Load the training dataset as ESPnet would configure it
dataset = ESPnetDataset(
    path_name_type_list=[("data/train/wav.scp", "speech", "sound"),
                         ("data/train/text", "text", "text")],
    preprocess_fn=None,
)

# Instantiate numel sampler with same config as training
sampler = NumelBatchSampler(
    batch_bins=4_000_000,
    shape_files=["data/train/speech_shape", "data/train/text_shape"],
    sort_in_batch="descending",
    sort_batch="descending",
)

# Measure token count per batch over first 1000 batches
token_counts = []
for batch_idx, indices in enumerate(sampler):
    if batch_idx >= 1000:
        break
    total_tokens = 0
    for idx in indices:
        _, text = dataset[idx]
        total_tokens += len(text["text"])   # output token count
    token_counts.append(total_tokens)

token_arr = np.array(token_counts)
print(f"Batch token count statistics over first 1000 batches:")
print(f"  min:    {token_arr.min():,}")
print(f"  5th %:  {np.percentile(token_arr, 5):,.0f}")
print(f"  median: {np.median(token_arr):,.0f}")
print(f"  95th %: {np.percentile(token_arr, 95):,.0f}")
print(f"  max:    {token_arr.max():,}")
print(f"  std/mean (CV): {token_arr.std()/token_arr.mean():.3f}")
# Output (Westfield customer service corpus):
#   min:      3,241
#   5th %:    5,819
#   median:  11,480
#   95th %:  18,673
#   max:     21,034
#   std/mean (CV): 0.618  ← CV > 0.5 → high gradient magnitude variance

# Monitor gradient norm during training to confirm
# Add to ESPnet trainer callback (or custom training loop):
import torch

def log_gradient_norm(model):
    total_norm = 0.0
    for p in model.parameters():
        if p.grad is not None:
            param_norm = p.grad.data.norm(2)
            total_norm += param_norm.item() ** 2
    return total_norm ** 0.5

# Gradient norm log (excerpt from Westfield training run):
# Step 1000: loss=18.42, grad_norm=2.84, batch_tokens=18,420
# Step 1001: loss= 5.93, grad_norm=0.91, batch_tokens= 5,930   ← 3.1× drop
# Step 1002: loss=16.71, grad_norm=2.57, batch_tokens=16,710
# Step 1003: loss= 7.14, grad_norm=1.09, batch_tokens= 7,140   ← 2.4× drop
# Pattern: gradient norm swings 2-3× with each batch due to token count variance

Fix: normalize loss by batch token count to decouple gradient magnitude from batch composition

The fix is to normalize the total batch loss by the number of output tokens in the batch before calling loss.backward(). This converts ESPnet2’s default summed loss into a per-token average loss, so the gradient magnitude reflects the per-token learning signal rather than how many tokens happened to land in the current batch. With token normalization, a 19,280-token short batch produces the same per-token loss ≈ 1.0 and the same gradient magnitude as a 6,095-token long batch at the same per-token cross-entropy. The 3.2× gradient variance collapses to a coefficient of variation below 0.1, the training loss oscillation disappears, and val_WER converges from 0.19 to 0.11 by epoch 50.

# Fix: normalize ESPnet2 CTC + attention loss by batch token count
# ESPnet 202301.x, Python 3.11, PyTorch 2.1.x

# Option A: modify the ESPnet2 model's forward() to return per-token loss
# (recommended — keeps loss magnitude consistent regardless of batch composition)

# In espnet2/asr/espnet_model.py (or a subclass), modify forward():

class ESPnetASRModel_TokenNorm(ESPnetASRModel):
    def forward(self, speech, speech_lengths, text, text_lengths, **kwargs):
        # --- standard ESPnet2 forward pass ---
        loss, stats, weight = super().forward(
            speech, speech_lengths, text, text_lengths, **kwargs
        )

        # Count output tokens in this batch
        # text_lengths: (batch,) tensor of token counts per utterance
        batch_tokens = text_lengths.sum().float()

        # Normalize loss by token count
        # Before: loss = sum(ctc_loss_per_token + att_loss_per_token)
        # After:  loss = mean(ctc_loss_per_token + att_loss_per_token)
        loss_normalized = loss / batch_tokens

        # Update stats for logging
        stats["batch_tokens"] = batch_tokens.item()
        stats["loss_per_token"] = loss_normalized.item()

        return loss_normalized, stats, weight

# Option B: use ESPnet2 config parameter (if available in your version)
# Some ESPnet2 versions support normalizing loss via model_conf:
# model_conf:
#     ctc_weight: 0.3
#     length_normalized_loss: true   # ← divides by token count before backward
# Check your ESPnet version's ESPnetASRModel for this parameter.

# Option C: switch to batch_type=length for more homogeneous batches
# batch_type: length
# batch_bins: 300000   # target: ~300s of audio per batch
# Sorts utterances by duration before batching → more similar token counts
# Trade-off: slightly lower GPU utilization vs numel; gradient variance CV ≈ 0.15

# Results after fix (Option A — token normalization):
# val_WER after 50 epochs: 0.11 (down from 0.19)
# Training loss oscillation: ±0.08 nats (down from ±4.0 nats)
# Gradient norm CV: 0.08 (down from 0.62)
# Training curve: smooth convergence visible from epoch 20

# ---- Hour breakdown ----
# Triage:     0.25h — val_WER plateau 0.19; training loss oscillates ±4.0 nats;
#             gradient norm varies 3× between consecutive steps
# Root cause: 0.5h — batch_type=numel with 5s–8min utterances;
#             short batches 19K tokens vs long batches 6K tokens;
#             summed loss = 3.2× higher for short batches; gradient magnitude variance
# Fix:        0.5h — normalize loss by batch_tokens in ESPnetASRModel.forward();
#             loss = total_loss / text_lengths.sum()
# Validation: 0.25h — val_WER 0.19→0.11; gradient CV 0.62→0.08; 1.5h total

Related retainer posts

Track retainer hours without a client portal

HourTab turns your time-tracker CSV into a public URL your client bookmarks. No login. No portal. CSV in, URL out.

See pricing →