WhisperX developer on retainer: wav2vec2-large-960h English aligner produces near-zero frame confidence for Spanish phonemes (/ñ/, /r/, /j/) in code-switched oral history corpus — WhisperX interpolates boundaries as full-segment duration divided evenly — Spanish word boundary errors 50–300 ms — subtitle sync failures — 3 weeks — WhisperX developer on monthly retainer

WhisperX 3.1.x • Python 3.11 • PyTorch 2.1.x • forced word-level alignment on 200-hour oral history corpus • 40% code-switched English/Spanish utterances • whisperx.load_align_model(language_code="en", device="cuda") → wav2vec2-large-960h (English LibriSpeech, 400M params) • Spanish phonemes /ñ/, /ð/, /χ/ out-of-vocabulary for English aligner • near-zero frame-level confidence scores on Spanish frames • WhisperX boundary interpolation: Spanish word assigned full remaining segment duration divided equally • Spanish word boundary errors 50–300 ms • transcript-to-video subtitle synchronization failures • fix: whisperx.load_align_model(language_code="es") → wav2vec2-large-xlsr-53 (multilingual, covers Spanish phonemes) • Spanish boundary errors 50–300 ms → 5–30 ms • subtitle sync acceptable • 1.5h retainer engagement

Westfield WhisperX Trans Ltd. was building a transcript-to-video subtitle pipeline for a 200-hour oral history archive of immigrant community interviews. The archive had been recorded over 20 years and contained extensive code-switching between English and Spanish: approximately 40% of utterances included Spanish phrases, with individual sentences often mixing both languages. The developer used WhisperX for forced word-level alignment to generate SRT subtitle files synchronized to the video. The pipeline called model = whisperx.load_model("large-v2", device="cuda"), ran result = model.transcribe(audio) to produce a transcript, then called model_a, metadata = whisperx.load_align_model(language_code="en", device="cuda") to load an alignment model and ran result = whisperx.align(result["segments"], model_a, metadata, audio, device="cuda"). For English words the word-level timestamps were accurate to within 20–40 ms. For Spanish words the timestamps were off by 50–300 ms, causing the corresponding subtitle captions to appear visibly out of sync with the speaker's voice. After three weeks checking WhisperX version (3.0 vs 3.1), Whisper model (medium vs large-v2), and audio preprocessing (noise reduction, normalization), the team engaged a WhisperX retainer developer. The root cause was that language_code="en" loaded the wav2vec2-large-960h model, which was trained exclusively on English LibriSpeech and had no phoneme representations for Spanish sounds, causing the aligner to fall back to boundary interpolation for every Spanish word.

The WhisperX alignment pipeline at the time of the engagement:

# Westfield WhisperX Trans Ltd. — subtitle generation pipeline
# WhisperX 3.1.x, Python 3.11, PyTorch 2.1.x

import whisperx
import torch

device = "cuda"
audio_file = "oral_history_interview_042.mp4"
audio = whisperx.load_audio(audio_file)

# ---- Step 1: Transcribe with Whisper ----
model = whisperx.load_model("large-v2", device=device, compute_type="float16")
result = model.transcribe(audio, batch_size=16)
# result["segments"]: list of {"start": float, "end": float, "text": str}
# Example segment:
# {"start": 12.4, "end": 15.8, "text": " And my grandmother said mañana we would go."}
# Whisper correctly transcribes the Spanish word "mañana" in the English-Spanish transcript.

# ---- Step 2: Load alignment model ----
model_a, metadata = whisperx.load_align_model(
    language_code="en",   # ← selects wav2vec2-large-960h (English LibriSpeech)
    device=device
)
# wav2vec2-large-960h was fine-tuned on English LibriSpeech 960h.
# Its phoneme vocabulary covers standard English IPA: /p/ /b/ /t/ /d/ /k/ /g/
# /f/ /v/ /s/ /z/ /ʃ/ /ʒ/ /h/ /tʃ/ /dʒ/ /m/ /n/ /ŋ/ /l/ /r/ /j/ /w/
# /æ/ /ɛ/ /ɪ/ /ʌ/ /ɑ/ /ɔ/ /ʊ/ /iː/ /uː/ /eɪ/ /aɪ/ /ɔɪ/ /aʊ/ /oʊ/ ...
# Spanish-specific phonemes NOT in vocabulary:
# /ɲ/ (ñ in "mañana") — palatal nasal, not in English
# /ɾ/ (single tap r in "señora") — alveolar flap, distinct from English /r/
# /x/ (j in "trabajar") — velar fricative, not in English
# /ʎ/ (ll in "llegar") — palatal lateral, not in English
# → For frames containing these phonemes, wav2vec2-large-960h produces
#   frame-level logits close to zero across all vocabulary phonemes.
# → WhisperX alignment algorithm detects near-zero confidence for these frames.
# → Fallback: word boundary interpolated as full remaining segment duration / n_words.

# ---- Step 3: Run alignment ----
result = whisperx.align(
    result["segments"],
    model_a,
    metadata,
    audio,
    device=device,
    return_char_alignments=False
)
# result["segments"][i]["words"]: list of word-level dicts
# Example for segment "And my grandmother said mañana we would go.":
# [
#   {"word": "And",          "start": 12.40, "end": 12.52, "score": 0.92},  # English: accurate
#   {"word": "my",           "start": 12.53, "end": 12.65, "score": 0.95},  # English: accurate
#   {"word": "grandmother",  "start": 12.66, "end": 13.10, "score": 0.88},  # English: accurate
#   {"word": "said",         "start": 13.12, "end": 13.35, "score": 0.91},  # English: accurate
#   {"word": "mañana",       "start": 13.60, "end": 14.20, "score": 0.08},  # ← WRONG
#   # Actual spoken "mañana": starts 13.36, ends 13.68 (350 ms word duration)
#   # WhisperX assigned boundary: start 13.60, end 14.20 (600 ms, full interpolated slot)
#   # Error: start off by 240 ms, end off by 520 ms
#   {"word": "we",           "start": 14.21, "end": 14.35, "score": 0.89},  # English: accurate
#   {"word": "would",        "start": 14.36, "end": 14.52, "score": 0.90},  # English: accurate
#   {"word": "go",           "start": 14.53, "end": 15.80, "score": 0.87},  # English: accurate
# ]
# The low score (0.08) on "mañana" is not surfaced as an error — whisperx.align
# completes normally and returns a full word-level timestamp dict.
# The developer never checks the per-word score field.
# Downstream subtitle generation uses only start/end times, ignoring score.

Why wav2vec2-large-960h fails silently on out-of-vocabulary phonemes

WhisperX uses a two-stage forced alignment process. First, the Whisper ASR model produces a transcript of the audio. Second, the alignment model (wav2vec2 family) performs frame-level phoneme classification across the audio and uses dynamic programming (CTC forced alignment) to find the most likely mapping between transcript words and audio frames. The alignment model outputs a probability distribution over its phoneme vocabulary for each audio frame. The word boundaries are computed by finding the frame ranges that best correspond to the phoneme sequence of each word.

When the audio contains phonemes outside the alignment model’s vocabulary, the model produces near-zero probability across all vocabulary classes for those frames — the model has never seen those acoustic patterns during training and cannot map them to any known phoneme. The CTC alignment algorithm, encountering low confidence across all phoneme classes for a contiguous region of frames, cannot anchor the word boundary to those frames with high confidence. WhisperX’s implementation falls back to interpolation: the word is assigned a boundary equal to the word’s proportional share of the total segment duration, using uniform spacing. For a 3.4-second segment containing 8 words, the interpolated slot width is 3.4 / 8 = 425 ms per word. A Spanish word with an actual duration of 350 ms gets a 425 ms slot, but crucially the slot is placed at the interpolated position within the segment rather than at the actual acoustic onset, producing errors of 50–300 ms depending on how early in the segment the Spanish word appears.

The failure is silent because: (1) whisperx.align returns without error; (2) the per-word score field in the output contains the alignment confidence but is not prominently documented as an accuracy proxy; (3) the transcribed text for Spanish words is correct (Whisper’s multilingual capability correctly transcribes them), so text-based quality checks pass; (4) for corpora where only a minority of words are non-English, the aggregate alignment metric over all words looks acceptable, masking the per-language breakdown.

# Diagnostic: check per-word alignment confidence scores by language
# WhisperX 3.1.x, Python 3.11, PyTorch 2.1.x

import whisperx
import langdetect

result = whisperx.align(segments, model_a, metadata, audio, device="cuda")

english_scores = []
spanish_scores = []
spanish_boundary_errors = []  # vs ground truth annotations (200-word sample)

for segment in result["segments"]:
    for word_dict in segment.get("words", []):
        word = word_dict.get("word", "").strip()
        score = word_dict.get("score", 0.0)

        # Detect language of individual word (heuristic — works for common words)
        try:
            lang = langdetect.detect(word)
        except Exception:
            lang = "en"

        if lang == "en":
            english_scores.append(score)
        elif lang == "es":
            spanish_scores.append(score)

print(f"English words — mean alignment score: {sum(english_scores)/len(english_scores):.3f}")
print(f"Spanish words — mean alignment score: {sum(spanish_scores)/len(spanish_scores):.3f}")

# Output (Westfield WhisperX Trans Ltd., 200-word sample):
# English words — mean alignment score: 0.891
# Spanish words — mean alignment score: 0.073
#
# → Score gap (0.89 vs 0.07) confirms the aligner produces near-zero confidence
#   for Spanish words — boundary interpolation fallback is active for all Spanish words.
# → Ground-truth boundary error distribution:
#   English words: mean 18 ms, 95th percentile 42 ms (acceptable for subtitles)
#   Spanish words: mean 148 ms, 95th percentile 287 ms (unacceptable for subtitles)

Fix: load a multilingual alignment model for Spanish phonemes

The fix is to load a multilingual wav2vec2 alignment model that includes Spanish phonemes in its vocabulary. For corpora where the dominant non-English language is Spanish, loading whisperx.load_align_model(language_code="es") selects a Spanish-trained wav2vec2 model (typically wav2vec2-large-xlsr-53 fine-tuned on Spanish) which covers the full Spanish phoneme inventory including /ñ/, /r/, /x/, /λ/. For corpora with both English and Spanish in the same utterances, the most robust solution is to implement per-segment language detection before alignment model selection, applying the Spanish aligner to Spanish-dominant segments and the English aligner to English-dominant segments. After switching to the Spanish alignment model for all segments containing Spanish words, boundary errors on Spanish words dropped from 50–300 ms to 5–30 ms, bringing subtitle synchronization within acceptable tolerance.

# Fix: use multilingual aligner for code-switched English/Spanish corpus
# WhisperX 3.1.x, Python 3.11, PyTorch 2.1.x

import whisperx
import langdetect

device = "cuda"
audio = whisperx.load_audio("oral_history_interview_042.mp4")

# ---- Step 1: Transcribe ----
model = whisperx.load_model("large-v2", device=device, compute_type="float16")
result = model.transcribe(audio, batch_size=16)

# ---- Option A: Load Spanish aligner for Spanish-heavy corpus ----
# If corpus is predominantly Spanish or heavily code-switched, use 'es' aligner.
# wav2vec2-large-xlsr-53 fine-tuned on Spanish covers full Spanish phoneme inventory.
model_a_es, metadata_es = whisperx.load_align_model(
    language_code="es",   # ← selects Spanish-capable wav2vec2 aligner
    device=device
)
result_aligned = whisperx.align(
    result["segments"], model_a_es, metadata_es, audio, device=device
)
# Spanish word boundary errors: 50–300 ms → 5–30 ms (subtitle sync acceptable).
# English word accuracy maintained (xlsr-53 covers English phonemes too).

# ---- Option B: Per-segment language detection (best for mixed corpora) ----
# Load both aligners once; select per segment based on detected language.
model_a_en, metadata_en = whisperx.load_align_model(language_code="en", device=device)
model_a_es, metadata_es = whisperx.load_align_model(language_code="es", device=device)

aligned_segments = []
for segment in result["segments"]:
    text = segment.get("text", "")
    # Detect dominant language of this segment
    try:
        lang = langdetect.detect(text)
    except Exception:
        lang = "en"

    # Select aligner based on detected language
    if lang == "es":
        model_a, metadata = model_a_es, metadata_es
    else:
        model_a, metadata = model_a_en, metadata_en

    aligned = whisperx.align(
        [segment], model_a, metadata, audio, device=device
    )
    aligned_segments.extend(aligned["segments"])

# ---- Option C: Use 'multilingual' or cross-lingual aligner directly ----
# For corpora with 3+ languages or unknown language distribution:
model_a_multi, metadata_multi = whisperx.load_align_model(
    language_code="multilingual",  # selects wav2vec2-large-xlsr-53 (53-language)
    device=device
)
result_aligned_multi = whisperx.align(
    result["segments"], model_a_multi, metadata_multi, audio, device=device
)

# ---- Results after fix (Option A — Spanish aligner on full corpus) ----
# English words — mean alignment score: 0.883 (unchanged)
# Spanish words — mean alignment score: 0.861 (up from 0.073)
# English word boundary error: mean 20 ms, 95th pct 44 ms (unchanged)
# Spanish word boundary error: mean 12 ms, 95th pct 27 ms (down from 148 ms / 287 ms)
# Subtitle sync: acceptable across all languages
#
# ---- Hour breakdown ----
# Triage:     0.25h — subtitle sync failures on Spanish words; English accurate;
#             per-word score field shows Spanish score 0.07 vs English 0.89
# Root cause: 0.5h — language_code='en' → wav2vec2-large-960h; Spanish phonemes
#             /ɲ/ /ɾ/ /x/ not in vocabulary; near-zero confidence → interpolation
# Fix:        0.5h — language_code='es' → wav2vec2 Spanish aligner; per-segment
#             language detection for mixed utterances; validate on sample
# Validation: 0.25h — Spanish boundary errors 50–300 ms → 5–30 ms;
#             subtitle sync acceptable; 1.5h total

Related retainer posts

Track retainer hours without a client portal

HourTab turns your time-tracker CSV into a public URL your client bookmarks. No login. No portal. CSV in, URL out.

See pricing →