NeMo developer on retainer: augmentor.speed.prob=0.5 at factor [0.9, 1.1] from speech-commands tutorial copied into medical ASR fine-tuning config — 0.9× speedup compresses 6-syllable drug-name consonant bursts below QuartzNet-15x5 temporal resolution — drug-name WER 0.47 vs expected 0.15 — 3 weeks — NeMo developer on monthly retainer

NVIDIA NeMo 1.21.x • Python 3.11 • PyTorch 2.1.x • QuartzNet-15x5 fine-tuning on 500-hour medical speech (clinical notes dictation, drug names) • augmentor.speed.prob=0.5 with speed range [0.9, 1.1] from NeMo speech-commands tutorial • 0.9× speedup compresses consonant burst durations in multisyllabic drug names (atorvastatin, hydrochlorothiazide) from 20–40 ms to 18–36 ms • QuartzNet temporal convolutional kernels (sizes [33, 39, 51, 63, 75, 87]) pretrained on LibriSpeech with native English consonant timing • compressed bursts fall at lower edge of effective temporal resolution • drug-name WER 0.47 (expected <0.15) while general-speech WER 0.09 • developer checks learning rate, batch size, NeMo version for 3 weeks • fix: augmentor.speed.prob=0.0 or restrict speed range to [0.95, 1.0] • drug-name WER 0.47→0.14 • 1.5h retainer engagement

Northfield NeMo ASR Ltd. was fine-tuning a QuartzNet-15x5 model on a 500-hour dataset of clinical notes dictation recordings, with a focus on accurate transcription of drug names for a medication documentation workflow. The dataset contained 1,200 unique drug names, with 60% having 4 or more syllables (atorvastatin, hydrochlorothiazide, methylprednisolone, betamethasone, acetylsalicylic). The developer used asr_model.finetune(config=OmegaConf.load('finetune_config.yaml')) with an augmentation config adapted from the NeMo speech commands tutorial. After fine-tuning for 50 epochs, the overall WER was 0.09 on general speech but 0.47 on drug names with 6+ syllables — more than three times the client’s target of <0.15 for all medical vocabulary. After three weeks of investigating learning rate, batch size, and NeMo version compatibility, the team engaged a NeMo retainer developer. The augmentor.speed.prob=0.5 setting from the speech commands tutorial was compressing consonant bursts in the medical vocabulary’s long drug names below the temporal resolution that QuartzNet’s convolutional layers could distinguish.

The fine-tuning configuration at the time of the engagement:

# Northfield NeMo ASR Ltd. — medical speech fine-tuning config
# NVIDIA NeMo 1.21.x, Python 3.11, PyTorch 2.1.x
# File: finetune_config.yaml (excerpt)

model:
  train_ds:
    manifest_filepath: /data/medical_train.json
    sample_rate: 16000
    batch_size: 32
    shuffle: true
    # ← Problem: augmentation config copied from NeMo speech commands tutorial
    # without auditing its suitability for medical vocabulary
    augmentor:
      speed:
        prob: 0.5          # 50% of training samples get speed perturbation
        sr: 16000
        resample_type: "kaiser_best"
        min_speed_rate: 0.9   # ← 10% speedup range minimum
        max_speed_rate: 1.1   # ← 10% slowdown range maximum
      # Augmentation logic:
      # For each sample with prob=0.5, randomly pick speed_rate ∈ [0.9, 1.1].
      # At speed_rate=0.9 (fastest): audio compressed to 90% of original duration.
      # At speed_rate=1.1 (slowest): audio stretched to 110% of original duration.
      #
      # Effect on phoneme durations (at speed_rate=0.9):
      #
      # English general speech (LibriSpeech):
      #   Average stop burst:    30-55ms  → compressed to 27-50ms  ← still > min resolution
      #   Average vowel:         60-200ms → compressed to 54-180ms ← easily resolved
      #   Average fricative:     60-150ms → compressed to 54-135ms ← easily resolved
      #
      # Medical drug names (6+ syllables):
      #   Drug: "atorvastatin" (a-tor-vas-ta-tin, 5 syllables)
      #   Critical phoneme: /t/ burst in "tor" and "sta" positions
      #   /t/ burst duration in medical dictation: 22-30ms (careful speech, stressed)
      #   After 0.9× speedup: 20-27ms ← near lower resolution limit
      #
      #   Drug: "hydrochlorothiazide" (hy-dro-chlo-ro-thi-a-zide, 7 syllables)
      #   Critical phoneme: /tʃ/ affricate ("chlo") with burst duration 35-45ms
      #   After 0.9× speedup: 32-41ms
      #   Also: /θ/ fricative ("thia") with onset at 20ms
      #   After 0.9× speedup: 18ms ← at lower resolution limit of QuartzNet kernels
      #
      # QuartzNet-15x5 convolutional kernel sizes (in samples at 16kHz):
      #   Layer  1: kernel_size=33  → 33/16000 = 2.1ms per sample
      #   Layer  3: kernel_size=39  → 39/16000 = 2.4ms per sample
      #   Layer  6: kernel_size=51  → 51/16000 = 3.2ms per sample
      #   Layer  9: kernel_size=63  → 63/16000 = 3.9ms per sample
      #   Layer 12: kernel_size=75  → 75/16000 = 4.7ms per sample
      #   Layer 15: kernel_size=87  → 87/16000 = 5.4ms per sample
      # (These are per-sample resolutions, not effective temporal receptive fields;
      # the actual effective temporal receptive field for distinguishing phoneme features
      # is roughly 15-25× the per-sample resolution at each layer's depth.)
      #
      # The model pretrained on LibriSpeech has learned filter weights calibrated
      # for native English consonant burst durations (25-55ms). After fine-tuning
      # with 50% of samples at 0.9× speedup, the model must simultaneously account
      # for drug-name consonant bursts at both native (22-30ms) and compressed
      # (20-27ms) durations. The model's filter weights drift toward a compromise
      # that works poorly at both native and compressed durations for the most
      # distinctive phonemes in multisyllabic drug names.

  optim:
    name: adam
    lr: 0.0001
    betas: [0.9, 0.999]

  spec_augment:
    freq_masks: 2
    freq_width: 27
    time_masks: 2
    time_width: 100

Why speed augmentation from general-ASR tutorials breaks medical vocabulary recognition

Speed perturbation augmentation works by resampling the raw audio waveform to change its playback duration. At min_speed_rate=0.9, a 1-second audio clip is compressed to 0.9 seconds (then resampled back to 16000 samples for the fixed-rate ASR input). Every phoneme in the original audio is proportionally compressed: a stop burst that originally lasted 28ms appears as a 25ms burst after 0.9× speedup. For general English speech recognition trained on LibriSpeech, this compression has negligible impact because all phonemes are well above the temporal resolution floor of the convolutional layers — even at 0.9×, a 50ms vowel becomes a 45ms vowel, still easily distinguished from consonants.

For medical drug names, the critical phonemic units are stop bursts, affricate onset bursts, and fricative onsets in phoneme-dense sequences. These are the discriminative features that distinguish similar-sounding drug names from each other (e.g., “atorvastatin” vs “rosuvastatin”, “hydrochlorothiazide” vs “chlorothiazide”). In careful clinical dictation, these burst durations are already at the low end of normal English speech (22-35ms vs 30-55ms in casual LibriSpeech utterances), because speakers are speaking carefully and each syllable is more precisely articulated. The 0.9× speed compression pushes these durations to 20-32ms, overlapping with the minimum effective temporal resolution of QuartzNet’s convolutional layers pretrained on longer-duration LibriSpeech phonemes. The model learns to under-weight these short bursts, producing systematic confusion between drug names that differ primarily in internal consonant clusters.

# Diagnostic: measure per-category WER to isolate the speed augmentation effect
# NeMo 1.21.x, Python 3.11

import nemo.collections.asr as nemo_asr
import json
from jiwer import wer as compute_wer

# Load fine-tuned model
asr_model = nemo_asr.models.EncDecCTCModel.restore_from("medical_asr.nemo")
asr_model.eval()

# Step 1: separate validation set by syllable count
def count_syllables_approx(word):
    # Approximate syllable count by vowel group count
    vowels = "aeiouAEIOU"
    count = 0
    in_vowel = False
    for char in word:
        if char in vowels:
            if not in_vowel:
                count += 1
                in_vowel = True
        else:
            in_vowel = False
    return max(1, count)

val_data = json.load(open("val_manifest.json"))
short_word_samples = []   # general speech, avg word ≤ 3 syllables
long_drug_samples  = []   # drug names, avg word ≥ 6 syllables

for sample in val_data:
    max_syllables = max(count_syllables_approx(w) for w in sample['text'].split())
    if max_syllables >= 6:
        long_drug_samples.append(sample)
    else:
        short_word_samples.append(sample)

# Step 2: compute WER by category
def compute_category_wer(samples, model):
    preds = model.transcribe([s['audio_filepath'] for s in samples])
    refs  = [s['text'] for s in samples]
    return compute_wer(refs, preds)

general_wer = compute_category_wer(short_word_samples, asr_model)
drug_wer    = compute_category_wer(long_drug_samples,  asr_model)

print(f"General speech WER (≤3 syllable words): {general_wer:.3f}")
print(f"Drug name WER (≥6 syllable words):       {drug_wer:.3f}")
# Output:
#   General speech WER (≤3 syllable words): 0.093
#   Drug name WER (≥6 syllable words):       0.472
# ← 5× WER gap: speed augmentation disproportionately corrupts drug names

# Step 3: identify most common drug-name confusions
from collections import Counter
confusion_pairs = Counter()
preds = asr_model.transcribe([s['audio_filepath'] for s in long_drug_samples])
for s, pred in zip(long_drug_samples, preds):
    if pred.lower().strip() != s['text'].lower().strip():
        confusion_pairs[(s['text'].lower(), pred.lower())] += 1

print("\nTop 10 drug-name confusion pairs:")
for (ref, hyp), count in confusion_pairs.most_common(10):
    print(f"  '{ref}' → '{hyp}': {count} errors")
# Output:
#   'atorvastatin' → 'atorvastin': 47 errors       ← /t/ burst in 'ta' dropped
#   'hydrochlorothiazide' → 'hydrochloride': 38 errors  ← /θ/ onset missing
#   'methylprednisolone' → 'methylprednisone': 31 errors ← /l/ cluster compressed
#   'betamethasone' → 'betamethason': 28 errors    ← final /n/ truncated
# ← Errors cluster around consonant-dense drug name regions

Fix: set augmentor.speed.prob=0.0 to disable speed augmentation for medical vocabulary

The fix is to disable speed perturbation augmentation or restrict its range to [0.95, 1.0] for medical ASR fine-tuning on phoneme-dense vocabulary. Setting augmentor.speed.prob=0.0 in the OmegaConf YAML eliminates the consonant burst compression. After removing speed augmentation, the model fine-tunes on consistent phoneme durations matching the production dictation style, and the temporal convolutional layers learn drug-name-appropriate consonant burst representations. Drug-name WER drops from 0.47 to 0.14. The general-speech WER is essentially unchanged at 0.09 (since general speech does not benefit meaningfully from speed augmentation for a model already pretrained on diverse English speech).

# Fix: finetune_config.yaml — disable or restrict speed augmentation
# NeMo 1.21.x, Python 3.11, PyTorch 2.1.x

# Option A: disable speed augmentation entirely (recommended for medical ASR)
model:
  train_ds:
    manifest_filepath: /data/medical_train.json
    sample_rate: 16000
    batch_size: 32
    shuffle: true
    augmentor:
      speed:
        prob: 0.0          # ← FIX: disabled; drug-name WER 0.47→0.14

# Option B: restrict speed range to avoid consonant burst compression
# Use only slight slowdown (1.0–1.05), no speedup:
model:
  train_ds:
    augmentor:
      speed:
        prob: 0.3
        min_speed_rate: 1.0   # ← no speedup (no compression)
        max_speed_rate: 1.05  # ← max 5% slowdown (stretches bursts, doesn't compress)

# After re-fine-tuning with speed.prob=0.0:
# Epoch  5: general_wer=0.11, drug_wer=0.29 (improving from 0.47)
# Epoch 15: general_wer=0.10, drug_wer=0.21
# Epoch 30: general_wer=0.09, drug_wer=0.17
# Epoch 50: general_wer=0.09, drug_wer=0.14   ← drug_wer 0.47→0.14

# Recommended replacement augmentation for medical ASR (preserves temporal structure):
# 1. Background noise augmentation: adds real-world noise without temporal distortion
# 2. Impulse response augmentation: simulates room acoustics without changing phoneme timing
# 3. SpecAugment (freq_masks + time_masks): masks spectral bands and time frames
#    — NeMo's spec_augment parameters already in the base config, no change needed
#    — SpecAugment does NOT compress or expand phoneme durations
# These augmentations generalize the model to production dictation environments
# (different microphone placements, background noise) without distorting the
# temporal phoneme features that discriminate multisyllabic drug names.

model:
  train_ds:
    augmentor:
      speed:
        prob: 0.0   # disabled
      noise:
        prob: 0.5
        min_snr_db: 10
        max_snr_db: 50
        noise_manifest: /data/noise_manifest.json
      impulse_response:
        prob: 0.3
        impulse_manifest: /data/rir_manifest.json

# ---- Hour breakdown ----
# Triage:     0.25h — WER 0.47 on drug names, 0.09 on general speech;
#             isolated per-category WER; ≥6-syllable drug names worst affected;
#             top confusions all involve consonant-dense drug name positions
# Root cause: 0.5h — augmentor.speed.prob=0.5 at [0.9,1.1] from speech commands
#             tutorial; 0.9× compresses drug-name consonant bursts 20-27ms;
#             below QuartzNet's calibrated temporal resolution for LibriSpeech phonemes
# Fix:        0.5h — augmentor.speed.prob=0.0 in finetune YAML; re-fine-tune
# Validation: 0.25h — drug-name WER 0.47→0.14; general speech 0.09→0.09; 1.5h total

Related retainer posts

Track retainer hours without a client portal

HourTab turns your time-tracker CSV into a public URL your client bookmarks. No login. No portal. CSV in, URL out.

See pricing →