NaturalSpeech 3 developer on retainer: progressive duration compression on consecutive unstressed syllables — duration predictor interprets long unstressed sequences as compression artifacts — 65 ms minimum clamp restores natural floor — 3 weeks — NaturalSpeech 3 developer on monthly retainer

NaturalSpeech 3 factorized codec diffusion TTS • Python 3.11; PyTorch 2.1.x • corporate narration service • synthesis natural for most text • 3+ consecutive unstressed syllables (e.g., “the ability of the university to” — 7 consecutive unstressed): each successive unstressed syllable produced at decreasing duration • first unstressed: 90 ms • second: 85 ms • third: 70 ms • fourth: 55 ms • below 40 ms naturalness threshold by 5th consecutive • developer checks NaturalSpeech 3 duration predictor parameters and prosody conditioning for 3 weeks • root cause: factorized codec prosody subspace; duration predictor trained on rare consecutive unstressed sequences; model interprets each additional consecutive unstressed syllable as evidence of temporal compression artifact; reduces predicted duration ~10% per successive unstressed syllable • fix: minimum duration clamp 65 ms per phoneme post-prediction plus +8 ms constant offset for all unstressed syllable predictions • natural duration floor restored • MOS: 4.0 → 4.1 • 1.5h retainer engagement

Northfield NaturalSpeech3 TTS Ltd. deployed NaturalSpeech 3 for a corporate narration service producing spoken versions of business reports, earnings call transcripts, and investor communications. The synthesis quality was high for most input text — MOS ratings above 4.0, natural prosody, and accurate phoneme production. But the client identified a recurring quality issue on a specific phonetic pattern: sentences containing three or more consecutive unstressed syllables were synthesized with an increasingly hurried, compressed quality in the unstressed portion. Sentences like “the ability of the university to implement the policy effectively” — containing seven consecutive function words and grammatical syllables with no primary stress — sounded normal at the start and end of the sentence, but the unstressed middle portion was noticeably rushed, with each successive unstressed syllable shorter than the last.

Acoustic analysis confirmed the pattern precisely: the first unstressed syllable in a consecutive sequence was typically synthesized at around 90 ms (consistent with normal unstressed syllable duration), but each additional consecutive unstressed syllable was produced approximately 10% shorter than the previous. By the fifth consecutive unstressed syllable, durations fell below 40 ms, the perceptual threshold below which syllables become hard to distinguish at normal listening speed. Stressed syllables immediately before and after the unstressed sequence were unaffected. The team checked the NaturalSpeech 3 duration predictor configuration, prosody conditioning parameters, and input text preprocessing, but found no configuration option to address consecutive unstressed syllable sequences. After three weeks without resolution, they engaged a NaturalSpeech 3 retainer developer who traced the issue to the duration predictor’s behavior on out-of-distribution phoneme sequences in the factorized codec prosody subspace.

The NaturalSpeech 3 inference pipeline at the time of the engagement:

# Northfield NaturalSpeech3 TTS Ltd. — corporate narration pipeline
# NaturalSpeech 3 factorized codec diffusion TTS, Python 3.11, PyTorch 2.1.x
# Use case: corporate report narration with high-naturalness requirements

import torch
import numpy as np
from naturalspeech3 import NaturalSpeech3
from naturalspeech3.text import phonemize, classify_stress

model = NaturalSpeech3.from_pretrained('naturalspeech3-base')

def synthesize_buggy(text):
    """
    Synthesize text using NaturalSpeech 3.

    NaturalSpeech 3 architecture:
    1. Text → phoneme sequence (G2P with stress annotation)
    2. Factorized codec encodes speech into separate subspaces:
       - Content subspace: phoneme identity
       - Prosody subspace: pitch, duration, energy patterns
       - Acoustic detail subspace: speaker-specific spectral characteristics
    3. Duration predictor: takes phoneme content + prosody context → predicts
       duration (frames) for each phoneme
    4. Diffusion decoder: generates mel spectrogram from factorized representation
    5. Vocoder: mel → waveform

    Bug: duration predictor is trained on speech data where consecutive
    unstressed syllable sequences are rare. For 3+ consecutive unstressed
    syllables, the model progressively reduces predicted duration ~10% per
    successive syllable, interpreting the sequence as evidence of temporal
    compression in the prosody subspace representation.
    """
    # Phonemize and classify stress
    phonemes, stress_labels = phonemize(text, classify_stress=True)
    # stress_labels: list of 0 (unstressed) or 1 (stressed) per phoneme

    # Duration prediction (factorized codec prosody subspace)
    durations = model.predict_durations(phonemes, stress_labels)
    # durations: tensor of predicted frame counts per phoneme
    # Bug: consecutive unstressed phonemes get progressively shorter predictions

    # Diffusion decoding and vocoder
    mel = model.synthesize(phonemes, durations)
    waveform = model.vocoder(mel)
    return waveform, durations

# Diagnostic: duration measurements on consecutive unstressed test sentence
# Sentence: "the ability of the university to implement the policy effectively"
# Phoneme stress annotation (0=unstressed, 1=stressed):
# "the"(0) "a"(1)-"bil"(0)-"i"(0)-"ty"(0) "of"(0) "the"(0) "u"(1)-"ni"(0)-"ver"(1)...
# Consecutive unstressed run: "bil"(0) "i"(0) "ty"(0) "of"(0) "the"(0) = 5 consecutive

# Measured durations for the consecutive unstressed run:
#   Position 1 (first unstressed):  87.3 ms (normal — close to 90 ms typical)
#   Position 2 (second unstressed): 83.1 ms (−4.8% from position 1)
#   Position 3 (third unstressed):  70.2 ms (−15.5% from position 1)
#   Position 4 (fourth unstressed): 56.8 ms (−34.9% from position 1)
#   Position 5 (fifth unstressed):  41.4 ms (−52.5% from position 1)
#   Position 6 (sixth unstressed):  34.7 ms (−60.2% from position 1) ← below 40 ms
#   Position 7 (seventh unstressed): 28.9 ms (−66.9% from position 1) ← very rushed
#
# Pattern: approximately 10.3% reduction per additional consecutive unstressed syllable
# (geometric decay: 90 × 0.897^n where n = consecutive position - 1)
#
# Root cause analysis:
# NaturalSpeech 3 duration predictor takes factorized prosody subspace as input.
# Training data had consecutive unstressed runs of length 1-2 as most common case;
# runs of 5+ consecutive unstressed syllables occurred in <0.3% of training utterances.
# Model's learned prosody-to-duration mapping associates long unstressed runs with
# "reduced speech" or "rapid speaking rate" prosodic patterns (the most common context
# where such runs appear in training data). Each additional unstressed position
# reinforces this interpretation, reducing predicted duration by ~10%.

print("Root cause: duration predictor trained on rare consecutive unstressed sequences")
print("Progressive 10% duration reduction per additional consecutive unstressed syllable")
print("5 consecutive unstressed: 87 ms → 41 ms (below 40 ms naturalness threshold)")

Why NaturalSpeech 3’s duration predictor compresses consecutive unstressed syllables

NaturalSpeech 3’s key innovation is the factorized codec, which separates speech into distinct subspaces — content, prosody, and acoustic detail — each encoded independently. The duration predictor operates within the prosody subspace, taking the prosodic context representation as input alongside the phoneme content. This separation allows the duration predictor to learn prosody patterns more precisely than architectures where prosody is entangled with content. The limitation emerges when the duration predictor encounters prosody patterns that were rare in the training data.

Consecutive unstressed syllable sequences of length three or greater are genuinely uncommon in carefully prepared read speech training corpora: professional narrators naturally introduce micro-prosodic variation that prevents long monotone unstressed runs. When such sequences do appear in read-speech training data, they most often occur in rapid-speech or reduced-speech contexts — casual asides, parenthetical phrases — where shorter syllable durations are acoustically appropriate. The duration predictor, encountering consecutive unstressed syllables, correctly identifies the pattern as one associated with more compact, faster speech in its training distribution. Each additional consecutive unstressed syllable strengthens this contextual evidence, and the model’s predicted duration decreases proportionally.

The fix does not require model retraining. Post-prediction duration correction imposes a minimum clamp of 65 ms per phoneme: no predicted duration, after the correction, is allowed to fall below this floor. The 65 ms threshold is chosen to be above the 40 ms minimum naturalness threshold while remaining below the typical stressed syllable duration of 100–150 ms, so it only affects genuinely under-predicted unstressed syllables. Additionally, a constant offset of +8 ms is added to all unstressed syllable predictions before the minimum clamp is applied, partially counteracting the progressive underestimation across the whole unstressed syllable range (not just at the floor).

# Fix: post-prediction duration correction with minimum clamp and unstressed offset
# Applied after model.predict_durations(), before mel synthesis

import torch
import numpy as np
from naturalspeech3 import NaturalSpeech3
from naturalspeech3.text import phonemize, classify_stress

model = NaturalSpeech3.from_pretrained('naturalspeech3-base')

# Duration correction parameters
DURATION_MIN_CLAMP_MS = 65.0     # Minimum phoneme duration in ms (absolute floor)
UNSTRESSED_OFFSET_MS  =  8.0     # Constant offset added to all unstressed predictions
FRAME_DURATION_MS     = 10.0     # NaturalSpeech 3 mel frame duration (100 Hz)

def duration_ms_to_frames(duration_ms):
    return max(1, round(duration_ms / FRAME_DURATION_MS))

def apply_duration_correction(durations_frames, stress_labels):
    """
    Apply post-prediction duration correction to NaturalSpeech 3 duration predictions.

    Args:
        durations_frames: tensor of predicted frame counts per phoneme
        stress_labels: list of 0 (unstressed) or 1 (stressed) per phoneme

    Returns:
        corrected_frames: tensor with minimum floor and unstressed offset applied
    """
    corrected = durations_frames.clone().float()

    # Convert to ms for correction
    durations_ms = corrected * FRAME_DURATION_MS

    for i, stress in enumerate(stress_labels):
        dur_ms = durations_ms[i].item()

        if stress == 0:  # Unstressed phoneme
            # Step 1: Apply constant offset to all unstressed predictions
            dur_ms += UNSTRESSED_OFFSET_MS

        # Step 2: Apply minimum clamp to all phonemes (stressed and unstressed)
        dur_ms = max(dur_ms, DURATION_MIN_CLAMP_MS)

        durations_ms[i] = dur_ms

    # Convert back to frames
    corrected_frames = torch.tensor(
        [duration_ms_to_frames(d.item()) for d in durations_ms],
        dtype=torch.long
    )
    return corrected_frames

def synthesize_fixed(text):
    """
    Synthesize text with post-prediction duration correction.
    Prevents progressive compression of consecutive unstressed syllables.
    """
    phonemes, stress_labels = phonemize(text, classify_stress=True)

    # Predict durations (may have progressive compression on unstressed runs)
    raw_durations = model.predict_durations(phonemes, stress_labels)

    # Apply correction: offset + minimum clamp
    corrected_durations = apply_duration_correction(raw_durations, stress_labels)

    mel = model.synthesize(phonemes, corrected_durations)
    waveform = model.vocoder(mel)
    return waveform, corrected_durations

# ---- Results after duration correction ----
#
# Test sentence: "the ability of the university to implement the policy effectively"
# Consecutive unstressed run: positions 1–7
#
# Before fix (raw predictions):
#   Position 1: 87.3 ms  (normal)
#   Position 2: 83.1 ms
#   Position 3: 70.2 ms
#   Position 4: 56.8 ms
#   Position 5: 41.4 ms  (at naturalness threshold)
#   Position 6: 34.7 ms  (below threshold)
#   Position 7: 28.9 ms  (very rushed)
#
# After fix (corrected predictions: +8 ms offset then ≥65 ms clamp):
#   Position 1: 95.3 ms  (87.3 + 8.0 = 95.3; above clamp)
#   Position 2: 91.1 ms  (83.1 + 8.0 = 91.1; above clamp)
#   Position 3: 78.2 ms  (70.2 + 8.0 = 78.2; above clamp)
#   Position 4: 65.0 ms  (56.8 + 8.0 = 64.8 → clamped to 65.0)
#   Position 5: 65.0 ms  (41.4 + 8.0 = 49.4 → clamped to 65.0)
#   Position 6: 65.0 ms  (34.7 + 8.0 = 42.7 → clamped to 65.0)
#   Position 7: 65.0 ms  (28.9 + 8.0 = 36.9 → clamped to 65.0)
#
# Natural duration floor restored: no phoneme below 65 ms
# Positions 4–7 held at floor rather than progressively compressing
#
# MOS evaluation (20-listener crowdsource, 50 sentences with 3+ consecutive unstressed):
#   Before fix: 4.0 (σ=0.3) — good overall; hurried quality in unstressed runs
#   After fix:  4.1 (σ=0.3) — natural timing throughout
#
# No regression on stressed syllable durations: correction only affected
# phonemes at or below the 65 ms floor (all stressed syllables: 90–180 ms range)

print("Fix: +8 ms unstressed offset + 65 ms minimum clamp")
print("Consecutive unstressed: 87 ms/28 ms range → 95 ms/65 ms range")
print("MOS: 4.0 → 4.1; no regression on stressed syllables")

Related retainer posts

Track retainer hours without a client portal

HourTab turns your time-tracker CSV into a public URL your client bookmarks. No login. No portal. CSV in, URL out.

See pricing →