FastSpeech 2 developer on retainer: comma boundaries produce no pause — duration predictor assigns 1-frame (12 ms) duration to mid-sentence commas — LibriTTS training corpus has near-zero comma pause durations in forced alignments — duration predictor learns corpus statistics — 3 weeks — FastSpeech 2 developer on monthly retainer

FastSpeech 2 non-autoregressive TTS • ESPnet reference implementation • Python 3.11; PyTorch 2.1.x • trained on LibriTTS (960h) • IVR customer service application • comma boundaries produce 5–8 ms pause (perceptually absent) • period boundaries correct at 150–200 ms • developer checks text normalization, phoneme mapping, punctuation handling, Adam optimizer for 3 weeks • root cause: LibriTTS MFA forced alignments assign 1-frame (12 ms) duration to mid-sentence comma tokens; duration predictor learns corpus statistics; corpus reality = speakers do not pause at commas; fix: post-process duration predictor output to insert 8-frame (93 ms) minimum pause when comma or semicolon token appears in phoneme sequence • comma pause 5–8 ms → 93–100 ms • listener preference test: 76% post-processed vs 24% unprocessed • MOS unchanged at 4.2 • 1.5h retainer engagement

Westfield FastSpeech2 TTS Ltd. deployed a FastSpeech 2 model (ESPnet implementation, LibriTTS 960h training) as the synthesis backbone for a customer service IVR system. The IVR reads policy summaries, pricing tables, and terms-and-conditions excerpts to callers — content with heavy mid-sentence comma usage. The base FastSpeech 2 model produced accurate text reproduction and natural-sounding prosody on simple declarative sentences. When the team ran an A/B test comparing FastSpeech 2 against their legacy Tacotron 2 system, they noticed a consistent prosody gap: FastSpeech 2 produced no perceptible pause at comma boundaries inside long sentences. A sentence like “Your plan includes unlimited data, free roaming in 40 countries, and 24/7 customer support” sounded rushed in the FastSpeech 2 output — the commas were acoustically absent, while Tacotron 2 produced a natural 80–120 ms pause at each comma.

Sentence-ending periods were correct: FastSpeech 2 produced a 150–200 ms pause at every full stop. Only mid-sentence punctuation — commas, semicolons — was affected. The team spent three weeks checking text normalization (whether commas were being stripped before G2P), the phoneme mapping (whether commas mapped to a silence token or were discarded), the forced aligner configuration (MFA acoustic model, dictionary), and the Adam optimizer learning rate schedule. None of these investigations located the root cause. They engaged a FastSpeech 2 retainer developer who identified the problem in the first 30 minutes by inspecting the MFA forced alignment output on the LibriTTS training data: the ground-truth duration labels for comma tokens were overwhelmingly 1-frame (12 ms at the 86 Hz mel frame rate), because LibriTTS speakers naturally do not pause at mid-sentence commas when speaking unrehearsed.

The FastSpeech 2 pipeline at the time of the engagement:

# Westfield FastSpeech2 TTS Ltd. — FastSpeech 2 inference pipeline
# ESPnet FastSpeech 2, Python 3.11, PyTorch 2.1.x
# Trained on LibriTTS 960h + fine-tuned on domain IVR data

import numpy as np
from espnet2.bin.tts_inference import Text2Speech

# ---- G2P and phoneme sequence construction ----
# Input: "Your plan includes unlimited data, free roaming in 40 countries."
# G2P front-end maps text → phonemes + punctuation tokens:
#   ["Y", "UH0", "R", " ", "P", "L", "AE1", "N", " ", ..., "D", "EY1", "T", "AH0",
#    ",",   ← comma token: mapped to SP (silence phoneme)
#    " ", "F", "R", "IY1", " ", ...]
#
# SP (silence phoneme) is included in the phoneme vocabulary.
# The duration predictor assigns a predicted duration to each phoneme, including SP.

# ---- Duration predictor (the bug) ----
# Duration predictor architecture:
#   2-layer 1D CNN → 2-layer transformer → linear projection → softplus → scalar duration
# Trained with MSE loss against ground-truth durations from MFA forced alignment.
#
# LibriTTS MFA forced alignment statistics for comma (SP) tokens:
#   Mean duration: 1.1 frames (at 86 Hz mel frame rate = 12.8 ms)
#   Median duration: 1 frame (11.6 ms)
#   95th percentile: 2 frames (23.3 ms)
#   Maximum: 8 frames (93 ms) — rare cases where speaker paused
#
# MFA assigns short durations to commas because LibriTTS speakers do not pause:
#   "naturally spoken audiobook reading → no deliberate comma pauses"
#   Period (SP at sentence end): Mean 14.2 frames (165 ms) ← learned correctly
#   Comma (SP mid-sentence):     Mean 1.1 frames (12.8 ms)  ← too short
#
# Duration predictor inference for the test sentence:
#   phoneme "," (SP):  predicted duration = 1 frame (12 ms) ← the bug
#   phoneme "." (SP):  predicted duration = 13 frames (151 ms) ← correct

# ---- FastSpeech 2 inference without post-processing (bug) ----
def synthesize_buggy(text, model):
    # model returns phonemes, durations, pitch, energy, mel, wav
    output = model(text)
    phonemes = output["phonemes"]     # list of phoneme strings
    durations = output["durations"]   # shape: (n_phonemes,) — frames per phoneme
    # durations[i] = 1 for comma SP tokens → 12 ms pause
    # durations[i] = 13 for period SP tokens → 151 ms pause
    wav = output["wav"].numpy()
    return wav

# ---- Measurement: comma pause durations ----
# Test set: 50 IVR sentences with ≥2 commas each
# Method: extract waveform energy envelope; find near-silence regions at comma positions
#
# Results (before fix):
#   Mean comma pause:  7 ms (range: 5–11 ms)
#   Mean period pause: 162 ms (range: 148–181 ms) ← correct
#   Listener test: "comma pause audible?" → 3/20 listeners (15%) detected pause at comma
#   Listener test (Tacotron 2 baseline): 18/20 (90%) detected natural comma pause
print("Root cause: LibriTTS MFA comma SP tokens have 1-frame ground-truth duration")
print("Duration predictor learns corpus statistics; commas produce 12 ms pause at inference")
print("Period SP tokens correctly learned at 165 ms (sentence boundaries have natural pauses)")

Why the FastSpeech 2 duration predictor assigns near-zero duration to mid-sentence commas

FastSpeech 2’s duration predictor is a regression model trained to predict the number of mel frames each phoneme should occupy, using ground-truth durations derived from forced phoneme-to-audio alignment (Montreal Forced Aligner or similar). The ground-truth durations are a statistical reflection of how long each phoneme was actually held in the training recordings — they are not a linguistic prescription for how long pauses “should” last in well-paced speech. In naturally spoken audiobook recordings like LibriTTS, readers do not consistently pause at mid-sentence commas: a comma signals syntactic structure to the reader’s eye but rarely produces a measurable silence in fluent reading. MFA aligns the audio to the phoneme sequence and assigns the silence phoneme (SP) for comma tokens a duration that accurately reflects the acoustic reality: typically 1–2 frames (12–23 ms) in LibriTTS.

Sentence-ending periods are different: in LibriTTS, readers consistently produce a breath and a brief silence at sentence boundaries, and MFA assigns sentence-final SP tokens a mean duration of 14 frames (165 ms). The duration predictor learns this asymmetry faithfully — periods get long pauses; mid-sentence commas get near-zero pauses — because the training data consistently shows this pattern. At inference time on IVR content (which is formal, structured, and benefits from deliberate comma pauses for listener comprehension), the duration predictor produces prosodically incorrect output that reflects LibriTTS speaking style rather than the application’s naturalness requirements.

# Fix: post-process duration predictor output to enforce minimum pause on punctuation tokens
# FastSpeech 2, ESPnet, Python 3.11, PyTorch 2.1.x

import numpy as np

# ---- Punctuation pause configuration ----
# Mel frame rate: 22,050 Hz / 256 hop_length = 86.1 frames/second
# 1 frame ≈ 11.6 ms

PUNCTUATION_MIN_FRAMES = {
    ",":  8,   # comma:        8 frames ≈  93 ms
    ";":  9,   # semicolon:    9 frames ≈ 104 ms
    ":":  8,   # colon:        8 frames ≈  93 ms
    # Note: period / "." / "?" / "!" are already correct — do not override
}

def postprocess_durations(phonemes, durations, punct_min_frames=PUNCTUATION_MIN_FRAMES):
    """
    Override duration predictor output for punctuation SP tokens.
    phonemes: list of str, phoneme symbols including "," ";" ":" as punctuation markers
    durations: np.ndarray shape (n_phonemes,), predicted durations in mel frames
    Returns: np.ndarray, durations with punctuation minimums applied
    """
    durations = durations.copy().astype(float)
    for i, ph in enumerate(phonemes):
        if ph in punct_min_frames:
            min_dur = punct_min_frames[ph]
            if durations[i] < min_dur:
                durations[i] = min_dur
    return durations.astype(int)

# ---- Updated inference pipeline (fix applied) ----
def synthesize_fixed(text, model):
    # Get raw model output including phonemes and predicted durations
    raw = model(text, return_intermediates=True)
    phonemes = raw["phonemes"]
    durations = raw["durations"].numpy()

    # Apply punctuation minimum duration post-processing
    durations_fixed = postprocess_durations(phonemes, durations)

    # Re-synthesize with corrected durations
    # ESPnet exposes a duration override mechanism at inference time
    wav = model.decode_from_durations(
        phonemes=phonemes,
        durations=durations_fixed,
        pitch=raw["pitch"].numpy(),
        energy=raw["energy"].numpy(),
    )
    return wav.numpy()

# ---- Alternative: fix at training time (insert minimum duration in ground-truth labels) ----
# Option A: post-processing at inference (above) — fastest to deploy; no retraining
# Option B: add minimum-pause constraint to MFA post-processing during data prep:
#   for each comma SP token with duration < 8 frames:
#       set duration to 8 frames (stretch the SP interval in forced alignment)
#   then retrain duration predictor with corrected ground-truth labels
# Option B is preferred for production: the duration predictor learns correct behavior
# Option A is faster to validate before committing to full retraining

# ---- Results after fix (inference post-processing, Option A) ----
# Test set: 50 IVR sentences with ≥2 commas each
#
# Comma pause duration:
#   Before fix:  7 ms  (range: 5–11 ms)
#   After fix:   97 ms (range: 93–104 ms) ← clamped to 8 frames minimum
# Period pause duration:
#   Before fix:  162 ms (unchanged, periods not affected by post-processing)
#   After fix:   162 ms (unchanged)
# MOS (50 utterances, 10 listeners):
#   Before fix:  4.2 ± 0.1
#   After fix:   4.2 ± 0.1 (unchanged — MOS test is not sensitive to comma pauses)
# Listener preference test (comma-heavy sentences only, A/B):
#   FastSpeech 2 with post-processing: 76% preferred (38/50 listeners)
#   FastSpeech 2 without post-processing: 24% preferred (12/50 listeners)
# IVR comprehension score (caller correctly answers question about content):
#   Before fix:  71%
#   After fix:   84%

# ---- Why not simply increase the SP token duration prediction via fine-tuning? ----
# The duration predictor could be fine-tuned on IVR domain data with manually added
# comma pauses (Option B). However:
#   1. IVR domain data is limited (< 2 hours of recordings available)
#   2. MFA forced alignment on manually paced IVR speech would be more accurate,
#      but re-recording studio data is expensive ($1,500–3,000 for 2h professional)
#   3. Post-processing is deterministic and interpretable: exactly 93 ms at every comma
#   4. Post-processing latency impact: ~0.3 ms per sentence (array copy + scan)
# Decision: deploy Option A immediately; plan Option B fine-tuning if A/B test validates

# ---- Hour breakdown ----
# Triage:     0.25h — comma pauses absent; period pauses correct; isolated to SP tokens
# Root cause: 0.5h — MFA forced alignment inspection: comma SP durations 1-frame in LibriTTS;
#             duration predictor trains on corpus statistics; learns commas = 12 ms;
#             period-final SP correctly learned at 165 ms
# Fix:        0.5h — postprocess_durations() inserted in inference pipeline;
#             min 8 frames on comma, semicolon, colon SP tokens
# Validation: 0.25h — comma pause 7 ms → 97 ms; listener preference 76%; MOS unchanged; 1.5h total
print("Fix complete: comma pause 7 ms → 97 ms; listener preference 76% post-processed")

Related retainer posts

Track retainer hours without a client portal

HourTab turns your time-tracker CSV into a public URL your client bookmarks. No login. No portal. CSV in, URL out.

See pricing →