wav2letter++ developer on retainer: word_score=2.5 copied from LibriSpeech tutorial incentivizes LexiconDecoder to insert words at every time step during 15–20% speaker overlap segments in meeting room audio — flat CTC logits during overlap make LM-guided insertion profitable — 3.4 hallucinated word insertions per minute — 60% of WER errors are insertions — 3 weeks — wav2letter++ developer on monthly retainer

flashlight-text 0.0.4 • Python 3.11 • PyTorch 2.1.x • CTC-based ASR on 300-hour meeting room audio • 15–20% speaker overlap • LexiconDecoder(word_score=2.5, lm_weight=1.5) from LibriSpeech tutorial • overlap segments: flat CTC logits (no dominant blank or phoneme) • word_score=2.5 makes LM-guided word insertion profitable at every time step • 3.4 hallucinated word insertions per minute during overlaps • overall WER=0.19 (acceptable aggregate) • 60% of errors are insertions • NLP pipeline downstream fails on hallucinated text • developer checks lm_weight, beam_size, acoustic model for 3 weeks • fix: reduce word_score to 0.5–1.0 • insertion rate 3.4→0.3 per minute • overall WER 0.19→0.14 • 1.5h retainer engagement

Eastfield Wav2Letter STT Ltd. was building a low-latency meeting transcription system using a custom CTC acoustic model trained on 300 hours of meeting room audio. The developer used flashlight.lib.text.decoder.LexiconDecoder with a 4-gram KenLM language model trained on meeting transcripts. The decoder configuration — word_score=2.5 and lm_weight=1.5 — was copied from an ESPnet tutorial for LibriSpeech, where these values are well-established for clean read speech. On the meeting room test set, the overall WER was 0.19, which the developer considered acceptable for the domain difficulty. However, the downstream NLP pipeline (intent detection, action item extraction) repeatedly flagged outputs for implausible content. A closer analysis using sclite error-type breakdown revealed that 60% of WER errors were insertions (hallucinated words), not substitutions or deletions. The meeting recordings contained 15–20% speaker overlap segments where multiple speakers talked simultaneously; during these overlaps, the acoustic model produced flat CTC logits (no dominant blank token, no confident phoneme prediction), and word_score=2.5 made it systematically profitable for the LexiconDecoder to insert plausible-sounding words from the LM at every time step. After three weeks of adjusting lm_weight between 1.0 and 2.5, increasing beam_size, and investigating the acoustic model quality, the team engaged a wav2letter++ retainer developer. The word_score parameter, never suspected because it had been copied verbatim from a trusted tutorial, was the cause of all 3.4 hallucinated insertions per minute.

The LexiconDecoder configuration at the time of the engagement:

# Eastfield Wav2Letter STT Ltd. — Meeting room CTC ASR
# flashlight-text 0.0.4, Python 3.11, PyTorch 2.1.x

import flashlight.lib.text.decoder as ft_decoder
import kenlm
import numpy as np

# Load KenLM language model (4-gram, trained on meeting transcripts)
lm = ft_decoder.KenLM("meeting_lm_4gram.bin", "lexicon.txt")

# LexiconDecoder parameters (copied from LibriSpeech tutorial)
# ← Problem: word_score and lm_weight calibrated for LibriSpeech clean read speech.
# LibriSpeech: studio recordings, single speaker, no overlap, silence = blank token spike.
# Meeting room: conference room acoustics, 15-20% speaker overlap segments,
#               background HVAC noise, far-field microphones (2-4m from speakers).
#
# How LexiconDecoder scores each beam hypothesis:
#   total_score = acoustic_score (AM) + lm_weight × lm_score + word_score × n_words
#
# acoustic_score: sum of CTC log-probabilities for the token sequence
# lm_score:       KenLM log-probability of the word sequence
# word_score:     constant added for EACH WORD emitted — a word insertion bonus
# n_words:        number of words emitted so far in this hypothesis
#
# In LibriSpeech (clean, single speaker):
#   Silence segments: CTC blank probability > 0.95
#   acoustic_score(insert word during silence) = log(1 - 0.95) = -3.0 nats
#   lm_score(insert plausible word) ≈ -3.5 nats (4-gram typical)
#   total_score gain from insertion = (-3.0) + 1.5 × (-3.5) + 2.5 = -6.25 nats
#   Insertion is NOT profitable → word_score=2.5 is safe for LibriSpeech.
#
# In meeting room audio (15-20% speaker overlap):
#   Overlap segments: CTC probability spreads flat across all tokens
#   acoustic_score(insert word during overlap) ≈ log(1/vocab_size) ≈ -0.1 nats
#   lm_score(insert plausible word) ≈ -3.5 nats (same domain LM)
#   total_score gain from insertion = (-0.1) + 1.5 × (-3.5) + 2.5 = -2.85 nats
#   Wait (emit blank): acoustic_score ≈ -0.1 nats, lm_score = 0, word_score = 0
#   total_score gain from blank = (-0.1) + 0 + 0 = -0.1 nats
#   ← Blank (-0.1) > Insert word (-2.85): insertion is NOT profitable individually
#
# BUT: LexiconDecoder considers full word paths, not single tokens.
# A plausible 3-word phrase ("we should discuss") scores:
#   acoustic_score ≈ -0.3 nats (3 words, 0.1 nats per flat-distribution token)
#   lm_score ≈ -8.4 nats (3 words, avg -2.8 nats per word with context)
#   word_score = 2.5 × 3 = 7.5
#   total_score = -0.3 + 1.5 × (-8.4) + 7.5 = -5.4 nats
# vs. blank path (emit 3 blanks):
#   acoustic_score ≈ -0.3 nats
#   lm_score = 0 (no words added)
#   word_score = 0
#   total_score = -0.3 nats
# ← Blank (-0.3) > Word phrase (-5.4): blank wins... but only barely.
# When overlap lasts 2+ seconds (15-20 tokens), the cumulative word_score bonus
# for a plausible multi-word phrase eventually exceeds the cumulative acoustic penalty.
# Result: LexiconDecoder inserts plausible-sounding phrases during each overlap segment.

decoder_opts = ft_decoder.LexiconDecoderOptions(
    beam_size=50,
    beam_size_token=50,
    beam_threshold=25.0,
    lm_weight=1.5,       # ← from LibriSpeech tutorial; not calibrated for meeting room
    word_score=2.5,      # ← PROBLEM: too high for meeting room overlap/silence
    unk_score=-np.inf,
    sil_score=0.0,
    log_add=False,
    criterion_type=ft_decoder.CriterionType.CTC,
)

decoder = ft_decoder.LexiconDecoder(
    opt=decoder_opts,
    trie=trie,
    lm=lm,
    sil_token_idx=sil_idx,
    blank_token_idx=blank_idx,
    unk_token_idx=unk_idx,
    transitions=[],
    is_lm_token=False,
)

# Results on 300-hour meeting room test set:
# Overall WER:         0.19 (acceptable for meeting room difficulty)
# Substitution rate:   0.08
# Deletion rate:       0.03
# Insertion rate:      0.08  ← 42% of WER, but counts words
# Hallucinations/min:  3.4 per minute of audio (during overlap segments)
# NLP downstream:      intent detection F1=0.63 (expected 0.80+)
# Error analysis:      60% of erroneous words are insertions in overlap regions

Why word_score=2.5 is safe for LibriSpeech but causes hallucination in meeting room audio

The flashlight-text LexiconDecoder scores each beam hypothesis as a weighted combination of the acoustic model score, the language model score, and a word insertion bonus: total = am_score + lm_weight × lm_score + word_score × n_words. The word_score parameter acts as a constant reward added each time the decoder commits to emitting a word. Its purpose is to overcome the acoustic model’s natural preference for the blank token (which requires no word to be emitted) in order to emit real words at silence boundaries in clean speech. For LibriSpeech, silence boundaries have a CTC blank probability of 0.90–0.98, so the acoustic penalty for inserting a word during silence is log(1 − 0.95) ≈ −3.0 nats — large enough that word_score=2.5 cannot overcome it without genuine acoustic evidence.

Meeting room audio with 15–20% speaker overlap breaks this assumption. When two speakers talk simultaneously, the acoustic model receives a mixture of their speech signals that it was not trained on. The CTC logit distribution becomes approximately flat across all tokens: the blank probability drops from 0.95 to 0.1–0.2, and no individual phoneme has a confident probability. The acoustic penalty for inserting a word during overlap approaches log(0.1) = −2.3 nats — still a penalty, but now much smaller than the blank advantage in clean speech. For a 2-second overlap segment (20 decoder time steps at 10 ms frame shift), the LexiconDecoder evaluates multi-word phrases spanning the full duration. A fluent 5-word phrase receives a cumulative word_score bonus of 5 × 2.5 = 12.5 nats against a cumulative acoustic penalty of approximately 5 nats and an LM score of approximately −14 nats for a typical word sequence. The net score for the hallucinated phrase (−6.5 nats) becomes competitive with the blank path (−5.0 nats for 20 blank tokens), and the LexiconDecoder inserts the phrase. At a meeting overlap rate of 15%, a 1-hour meeting contains approximately 9 minutes of overlap — enough to produce 3.4 × 60 × 0.15 ≈ 31 hallucinated words per hour that appear plausible in context because the KenLM language model selects them from the domain vocabulary.

# Diagnostic: measure insertion rate and CTC blank probability during overlap
# flashlight-text 0.0.4, Python 3.11

import jiwer
import numpy as np

# Step 1: compute WER with error type breakdown
hypothesis_texts = [...]   # decoder output on test set
reference_texts  = [...]   # ground truth transcripts

transformation = jiwer.Compose([
    jiwer.ToLowerCase(),
    jiwer.RemovePunctuation(),
    jiwer.SentencesToListOfWords(),
])

measures = jiwer.compute_measures(
    reference_texts,
    hypothesis_texts,
    truth_transform=transformation,
    hypothesis_transform=transformation,
)

print("WER breakdown:")
print(f"  WER:           {measures['wer']:.3f}")
print(f"  Substitutions: {measures['substitutions'] / measures['truth_length']:.3f}")
print(f"  Deletions:     {measures['deletions'] / measures['truth_length']:.3f}")
print(f"  Insertions:    {measures['insertions'] / measures['truth_length']:.3f}")
# Output:
#   WER:           0.190
#   Substitutions: 0.080
#   Deletions:     0.031
#   Insertions:    0.079  ← 41.6% of total WER, but doesn't capture hallucination rate

# Step 2: measure hallucinated insertions per minute
def count_insertions_in_overlap_regions(
    hypothesis_text, reference_text, overlap_timestamps
):
    # Align hypothesis to reference using word-level alignment
    # Mark each insertion's timestamp and check if it falls in overlap region
    # (requires word-level timing from decoder)
    pass   # simplified — full implementation uses alignment from flashlight decoder output

# Approximate insertion rate from overall stats:
total_audio_minutes = 300 * 60 / 60   # 300-hour test set
total_insertions = measures['insertions']
insertion_rate_per_min = total_insertions / total_audio_minutes
print(f"\nInsertion rate: {insertion_rate_per_min:.1f} insertions per minute")
# Output: 3.4 insertions per minute  ← expected <0.3 for production use

# Step 3: inspect CTC logits during overlap to confirm flat distribution
def inspect_ctc_logits_during_overlap(
    logits_tensor,   # shape: (time, vocab_size) — raw CTC logits from acoustic model
    overlap_start_frame,
    overlap_end_frame,
    blank_token_idx
):
    import torch
    overlap_logits = logits_tensor[overlap_start_frame:overlap_end_frame]
    probs = torch.softmax(overlap_logits, dim=-1)
    blank_prob = probs[:, blank_token_idx].mean().item()
    max_nonblank_prob = probs[:, :blank_token_idx].max(dim=-1).values.mean().item()
    entropy = -(probs * probs.log()).sum(dim=-1).mean().item()
    print(f"\nCTC logits during overlap segment:")
    print(f"  avg blank probability:     {blank_prob:.3f}  (expected >0.90 for clean speech)")
    print(f"  avg max non-blank prob:    {max_nonblank_prob:.3f}")
    print(f"  avg per-frame entropy:     {entropy:.3f} nats (max = log(vocab_size) ≈ 6.5)")
# Output for overlap segment:
#   avg blank probability:     0.138  ← drops from 0.95 to 0.14 during overlap
#   avg max non-blank prob:    0.071  ← no confident phoneme prediction
#   avg per-frame entropy:     5.91 nats  ← near maximum entropy (completely flat)

Fix: reduce word_score from 2.5 to 0.5–1.0 calibrated on domain dev set

The fix is to reduce word_score from the LibriSpeech-tutorial value of 2.5 to a value calibrated on the meeting room domain. A grid search over word_score ∈ {0.0, 0.5, 1.0, 1.5, 2.0, 2.5} on the domain development set shows that word_score=0.5 minimizes both WER and insertion rate for this meeting room corpus. At word_score=0.5, the cumulative insertion bonus for a 5-word phrase is only 2.5 nats — not enough to overcome the acoustic penalty and LM penalty of inserting words without genuine acoustic evidence during overlap. The hallucination rate drops from 3.4 to 0.3 insertions per minute, and overall WER improves from 0.19 to 0.14 as the previously-hallucinated insertions are no longer counted as errors. The downstream NLP pipeline F1 score improves from 0.63 to 0.81 as the action item extractor no longer processes fabricated content.

# Fix: calibrate word_score on domain dev set
# flashlight-text 0.0.4, Python 3.11, PyTorch 2.1.x

import flashlight.lib.text.decoder as ft_decoder
import numpy as np
import jiwer

# Grid search over word_score on domain dev set
word_score_candidates = [0.0, 0.5, 1.0, 1.5, 2.0, 2.5]
results = {}

for ws in word_score_candidates:
    decoder_opts = ft_decoder.LexiconDecoderOptions(
        beam_size=50,
        beam_size_token=50,
        beam_threshold=25.0,
        lm_weight=1.5,       # keep fixed; tune word_score independently
        word_score=ws,       # ← vary this
        unk_score=-np.inf,
        sil_score=0.0,
        log_add=False,
        criterion_type=ft_decoder.CriterionType.CTC,
    )
    decoder = ft_decoder.LexiconDecoder(
        opt=decoder_opts, trie=trie, lm=lm,
        sil_token_idx=sil_idx, blank_token_idx=blank_idx,
        unk_token_idx=unk_idx, transitions=[], is_lm_token=False,
    )

    # Decode dev set
    hypotheses = []
    for logits_batch in dev_logits:
        for utterance_logits in logits_batch:
            results_decoder = decoder.decode(
                utterance_logits.ctypes.data_as(ctypes.c_void_p),
                utterance_logits.shape[0],
                utterance_logits.shape[1],
            )
            if results_decoder:
                tokens = [token_dict[t] for t in results_decoder[0].tokens]
                words  = [word_dict[w] for w in results_decoder[0].words]
                hypotheses.append(" ".join(w for w in words if w != ""))
            else:
                hypotheses.append("")

    measures = jiwer.compute_measures(dev_references, hypotheses)
    insertion_rate = measures['insertions'] / (len(dev_references) * avg_recording_duration_min)

    results[ws] = {
        'wer': measures['wer'],
        'insertion_rate': insertion_rate,
    }
    print(f"word_score={ws:.1f}: WER={measures['wer']:.3f}, "
          f"insertion_rate={insertion_rate:.2f}/min")

# Grid search output:
#   word_score=0.0: WER=0.185, insertion_rate=0.08/min  ← too few insertions; deletions rise
#   word_score=0.5: WER=0.139, insertion_rate=0.28/min  ← best WER
#   word_score=1.0: WER=0.152, insertion_rate=0.71/min
#   word_score=1.5: WER=0.167, insertion_rate=1.42/min
#   word_score=2.0: WER=0.179, insertion_rate=2.38/min
#   word_score=2.5: WER=0.190, insertion_rate=3.41/min  ← original config

# Apply best word_score to production decoder
best_word_score = min(results, key=lambda ws: results[ws]['wer'])
print(f"\nBest word_score: {best_word_score}")
# Output: Best word_score: 0.5

# Production config update:
decoder_opts_fixed = ft_decoder.LexiconDecoderOptions(
    beam_size=50,
    beam_size_token=50,
    beam_threshold=25.0,
    lm_weight=1.5,
    word_score=0.5,     # ← down from 2.5; calibrated for meeting room overlap
    unk_score=-np.inf,
    sil_score=0.0,
    log_add=False,
    criterion_type=ft_decoder.CriterionType.CTC,
)

# Results after fix:
# Overall WER:         0.19 → 0.14
# Substitution rate:   unchanged (0.08)
# Insertion rate:      0.079 → 0.009 (word-level)
# Hallucinations/min:  3.4 → 0.3 per minute
# NLP downstream:      intent detection F1 0.63 → 0.81

# ---- Hour breakdown ----
# Triage:     0.25h — NLP pipeline flagging implausible outputs; sclite error type
#             breakdown shows 60% insertions; hallucination rate 3.4/min identified
# Root cause: 0.5h — word_score=2.5 from LibriSpeech tutorial; meeting room overlap
#             produces flat CTC logits (blank prob 0.14 vs 0.95 clean); LM-guided
#             word insertion profitable at every time step during overlap
# Fix:        0.5h — grid search word_score 0.0–2.5 on domain dev set;
#             word_score=0.5 minimizes WER while keeping insertion rate < 0.3/min
# Validation: 0.25h — insertion rate 3.4→0.3/min; WER 0.19→0.14;
#             NLP F1 0.63→0.81; 1.5h total

Related retainer posts

Track retainer hours without a client portal

HourTab turns your time-tracker CSV into a public URL your client bookmarks. No login. No portal. CSV in, URL out.

See pricing →