XTTS developer on retainer: French nasal vowels denasalized when using English reference audio for French synthesis — XTTS v2 speaker embedding encodes English phonetic inventory — near-zero nasal airflow applied to French nasal vowels — 3 weeks — XTTS developer on monthly retainer

Coqui XTTS v2 (coqui/XTTS-v2) multi-lingual zero-shot TTS • Python 3.11; PyTorch 2.1.x • multilingual audiobook narration service • English reference audio for French synthesis • English narration natural; French narration: æcorrect speaker identity but French nasal vowels /ã/ /ε&slash;/ denasalized — pronounced as oral vowels instead of nasal • French text with native French reference sounds correct • developer checks XTTS v2 language settings, reference audio length (3s, 10s, 30s), XTTS model version for 3 weeks • root cause: XTTS v2 speaker embedding from English reference encodes near-zero nasal airflow as speaker characteristic; English has no phonemic nasal vowels; embedding suppresses velopharyngeal opening in French nasal vowel synthesis • fix: use native French reference audio for French synthesis • denasalization eliminated • 1.5h retainer engagement

Westfield XTTS TTS Ltd. built a multilingual audiobook narration service that adapts a single narrator’s voice to multiple languages using XTTS v2’s zero-shot voice cloning capability. The service collected a single high-quality English reference recording from each narrator and used it to synthesize that narrator’s voice in all 17 languages supported by XTTS v2. English narration sounded natural and preserved the narrator’s voice accurately. French narration, using the same English reference audio, produced the narrator’s voice at an acceptable speaker similarity — but every French nasal vowel was wrong. French words like enfant (/ãfã/), vin (/vε&slash;/), bon (/b&o;slash;/), and brun (/b⇾&slash;/) were produced as oral vowels without nasal resonance, making the French output sound like a foreign speaker without nasal phonemes in their inventory.

The team tried changing the XTTS v2 language code parameter, confirming it was set to fr. They tried different reference audio lengths — 3 seconds, 10 seconds, 30 seconds — with the same English reference speaker. All produced the same systematic denasalization. They tried a different XTTS v2 checkpoint. They checked whether XTTS v2 had a known French nasal vowel issue in its issue tracker. After three weeks, they engaged an XTTS retainer developer who identified the English reference audio phonetic inventory as the source of the cross-lingual phonetic contamination.

The XTTS v2 multilingual inference pipeline at the time of the engagement:

# Westfield XTTS TTS Ltd. — XTTS v2 multilingual voice cloning pipeline
# Coqui XTTS v2 (coqui/XTTS-v2), Python 3.11, PyTorch 2.1.x
# Use case: multilingual audiobook narration from single English reference audio

from TTS.api import TTS
import torch

# Initialize XTTS v2
tts = TTS("tts_models/multilingual/multi-dataset/xtts_v2")
tts.to("cuda")

# ---- The bug: English reference audio for all languages ----
#
# Reference audio: 15 seconds of English narration from the target narrator
# This reference audio is used for ALL 17 languages XTTS v2 supports.
#
# XTTS v2 speaker encoder architecture (GPT-based conditioned on reference audio):
#   The speaker encoder extracts a 512-dimensional speaker embedding from the reference audio.
#   This embedding encodes:
#     1. Voice identity features: fundamental frequency range, formant structure,
#        spectral envelope, speaking rate, prosodic style — these transfer well across languages.
#     2. Phonetic inventory features: the phonetic characteristics PRESENT in the reference
#        language — including which phonemes exist, their production characteristics,
#        and critically, which phonemes DO NOT exist and therefore have near-zero
#        activation in the embedding.
#
# English phonetic inventory: no nasal vowels.
#   English vowels are all oral — nasal airflow (velopharyngeal opening) is not
#   used contrastively for vowels in English. It may appear phonetically adjacent
#   to nasal consonants, but English has no phonemic /ɑ̃/, /ɛ̃/, /ɔ̃/, /œ̃/.
#
# English reference embedding therefore encodes near-zero nasal airflow for vowels.
#
# When XTTS v2 synthesizes French using this English embedding:
#   French nasal vowel /ɑ̃/ (enfant, restaurant, blanc):
#     Expected: low-mid open back unrounded vowel with velopharyngeal opening,
#               nasal resonance at 250–500 Hz formant region
#     Produced: near-zero nasal airflow (English embedding characteristic)
#               → oral /a/ instead of nasal /ɑ̃/
#
#   French nasal vowel /ɛ̃/ (vin, pain, main):
#     Expected: front mid unrounded vowel with nasal resonance
#     Produced: oral /ɛ/ or /e/ — denasalized
#
# Result: all French nasal vowels are systematically denasalized.
# French words sound like they were spoken by an English speaker
# who has never acquired French nasal phonemes.

english_reference = "narrator_english_15s.wav"

# English synthesis — works correctly (reference language matches synthesis language)
tts.tts_to_file(
    text="The old lighthouse stood at the edge of the cliff.",
    speaker_wav=english_reference,
    language="en",
    file_path="output_english.wav"
)

# French synthesis — nasal vowels denasalized (reference language mismatch)
tts.tts_to_file(
    text="L'enfant regardait le vin blanc dans le verre de son père.",
    # /ɑ̃/ in enfant, /ɛ̃/ in vin, /ɑ̃/ in blanc — all denasalized
    speaker_wav=english_reference,  # English reference — the bug
    language="fr",
    file_path="output_french_buggy.wav"
)
# Output: French text spoken with correct speaker identity but all nasal vowels
# pronounced as oral vowels: "enfant" → /efat/, "vin" → /ve/, "blanc" → /blak/
print("Bug: French nasal vowels denasalized — English reference encodes near-zero nasal airflow")

Why XTTS v2 speaker embedding encodes phonetic inventory characteristics

XTTS v2 uses a GPT-based architecture where synthesis is conditioned on a speaker embedding extracted from a reference audio clip. The design intention is that this embedding captures voice identity — the acoustic characteristics that make one speaker sound different from another. In practice, the embedding captures both voice identity and phonetic style: the statistical distribution of phonetic features present in the reference audio. When the reference audio contains no nasal vowels (because it is in English), the speaker encoder produces an embedding where the nasal airflow dimension is near-zero for all vowel tokens. This is not a bug in XTTS v2 — it is the model accurately learning that this particular speaker, in this particular reference recording, produced no nasal vowels. The model cannot know that this is because English lacks nasal vowels, not because the speaker anatomically cannot produce them.

The fix requires collecting reference audio in the target synthesis language. A native French reference recording from the same narrator — or even a short bilingual recording where the narrator reads a few French sentences — produces a speaker embedding that encodes French nasal vowel characteristics. With a French reference, XTTS v2 synthesizes French nasal vowels correctly while maintaining the narrator’s voice identity. As a secondary fix when language-matched reference audio cannot be collected, spectrogram post-processing can partially restore nasal vowel quality: a forced aligner identifies nasal vowel tokens in the synthesized output, and a 250–500 Hz shelving filter increases energy in the nasal resonance region for those tokens. This approach is less reliable than proper language-matched reference audio because it requires accurate forced alignment of the denasalized output and applies a fixed spectral correction rather than a phonetically accurate synthesis.

# Fix: use language-matched reference audio for each synthesis language
# Coqui XTTS v2 (coqui/XTTS-v2), Python 3.11, PyTorch 2.1.x

from TTS.api import TTS
import torch

tts = TTS("tts_models/multilingual/multi-dataset/xtts_v2")
tts.to("cuda")

# ---- Language-matched reference audio policy ----
#
# Reference audio collection protocol:
#   For each supported language, collect a 10–20 second reference recording
#   from the target narrator reading text in THAT LANGUAGE.
#   Language-specific phoneme requirements:
#     French: must contain at least 2 of /ɑ̃/, /ɛ̃/, /ɔ̃/, /œ̃/ (nasal vowels)
#     Spanish: must contain /ɾ/ and /r/ (flap and trill)
#     Portuguese (BR): must contain nasal vowels /ɐ̃/, /ẽ/
#     German: must contain /ʏ/, /ø/, /æ/ (front rounded and near-open)
#     Japanese: must contain pitch accent patterns (H-L and L-H transitions)

language_references = {
    "en": "narrator_english_15s.wav",
    "fr": "narrator_french_12s.wav",   # French reference — contains nasal vowels
    "es": "narrator_spanish_14s.wav",
    "de": "narrator_german_11s.wav",
    # ... other languages
}

def synthesize_multilingual(text, language, narrator_id="narrator_01"):
    """
    Synthesize text in target language using language-matched reference audio.
    Reference audio language must match synthesis language for phonetic accuracy.
    """
    reference_wav = language_references.get(language)
    if reference_wav is None:
        raise ValueError(
            f"No {language} reference audio registered for {narrator_id}. "
            f"Collect a {language} reference recording before synthesizing {language} content."
        )

    output_path = f"output_{narrator_id}_{language}.wav"
    tts.tts_to_file(
        text=text,
        speaker_wav=reference_wav,
        language=language,
        file_path=output_path
    )
    return output_path

# French synthesis with language-matched reference — nasal vowels correct
synthesize_multilingual(
    text="L'enfant regardait le vin blanc dans le verre de son père.",
    language="fr"
)
# Output: French text with correct nasal vowels:
# enfant: /ɑ̃fɑ̃/ ✓ (nasal /ɑ̃/ present)
# vin: /vɛ̃/ ✓ (nasal /ɛ̃/ present)
# blanc: /blɑ̃/ ✓ (nasal /ɑ̃/ present)

# ---- Phonetic validation for French synthesis ----
# After generating French output, validate nasal vowel presence before delivery.
# Using a French phoneme classifier (e.g., Allosaurus or wav2vec2-based French ASR):

def validate_french_nasal_vowels(audio_path, expected_text):
    """
    Validate that synthesized French audio contains expected nasal vowels.
    Returns proportion of nasal vowel tokens correctly nasalized.
    Uses French phoneme recognizer to detect /ɑ̃/, /ɛ̃/, /ɔ̃/, /œ̃/ presence.
    """
    # Load French phoneme recognizer (Allosaurus or Phonemizer + wav2vec2)
    # Identify tokens that should contain French nasal vowels in expected_text
    # Check that corresponding audio segments have nasal resonance at 250–500 Hz
    # Return: nasal_accuracy (0.0 to 1.0)
    pass  # implement with allosaurus.app.predict_lang(audio_path, 'fra')

# ---- Results after fix ----
#
# French nasal vowel accuracy (proportion correctly nasalized):
#   English reference (buggy):  0.00  (0/10 nasal vowel tokens correctly nasalized)
#   French reference (fixed):   0.94  (94% nasal vowels correctly produced)
#
# Speaker identity (ECAPA-TDNN cosine similarity to English reference speaker):
#   English reference synthesis:  0.82  (English baseline)
#   French reference synthesis:   0.78  (similar; identity maintained across reference change)
#
# MOS (mean opinion score, French native speakers, 20 utterances):
#   English reference (buggy):  2.1 ± 0.3  (strongly unnatural — no nasal vowels)
#   French reference (fixed):   4.0 ± 0.2  (natural-sounding French)
#
# Bilingual reference (10s English + 10s French concatenated):
#   Nasal vowel accuracy: 0.87  (improvement over English-only; less than French-only)
#   Recommendation: language-specific references preferred over bilingual concatenation

# ---- Why not fine-tune XTTS on bilingual data? ----
# Fine-tuning XTTS v2 on bilingual English-French speaker data to decouple
# speaker identity from phonetic inventory is possible but expensive:
#   - Requires 1–2 hours of bilingual recordings per speaker
#   - Fine-tuning takes 24–48h on a single A100
#   - For a production service with many narrators, per-language reference audio
#     collection (10–20 seconds per narrator per language) is far more practical.
# Fine-tuning is appropriate for a fixed narrator catalogue of 5–10 speakers;
# reference audio policy is appropriate for a dynamic narrator marketplace.

# ---- Hour breakdown ----
# Triage:     0.25h — French nasal vowels denasalized in all outputs;
#             English outputs correct; systematic phoneme-class failure suggests
#             reference audio phonetic inventory mismatch
# Root cause: 0.5h — XTTS v2 speaker embedding analysis;
#             English phonetic inventory has no nasal vowels;
#             embedding encodes near-zero nasal airflow;
#             French synthesis suppresses velopharyngeal opening;
#             French reference confirms correct nasal vowel production
# Fix:        0.5h — French native reference audio collection (12 seconds);
#             language-matched reference policy documentation;
#             reference validation protocol for nasal vowel languages
# Validation: 0.25h — 94% nasal vowel accuracy with French reference;
#             speaker identity maintained (cosine sim 0.78); MOS 2.1 → 4.0; 1.5h total
print("Fix complete: French nasal vowels correct with French reference; MOS 2.1 → 4.0")

Related retainer posts

Track retainer hours without a client portal

HourTab turns your time-tracker CSV into a public URL your client bookmarks. No login. No portal. CSV in, URL out.

See pricing →