Tortoise TTS developer on retainer: rare proper nouns mispronounced 20% of the time with fast preset — CLVP reranking requires N=512 to surface correct pronunciations — top-4 autoregressive candidates all share the same mispronunciation — 3 weeks — Tortoise TTS developer on monthly retainer

Tortoise TTS (neonbjb/tortoisetesting) auto-regressive TTS with diffusion vocoder • Python 3.11; PyTorch 2.1.x • audiobook production service • ‘standard’ preset (N=512 CLVP candidates): 45 s per sentence (acceptable for batch) • ‘fast’ preset (N=4): 12 s per sentence • fast preset: rare proper nouns mispronounced 20–30% of runs (th→d in “Featherstone”; wrong stress “RAvenscroft” instead of “raVENscroft”) • standard preset: correct • developer checks voice conditioning clips, random seed, CLVP reranking step count for 3 weeks • root cause: CLVP N=4 insufficient for rare proper nouns; autoregressive model assigns high probability to incorrect pronunciation; top-4 candidates all share same mispronunciation; N=512 surfaces low-probability correct pronunciation for CLVP to select • fix: spaCy NER detects sentences with rare proper nouns; N=32 candidates for those sentences; N=4 for all others • mispronunciation rate: 20% → 2% • latency: 12 s common text; 28 s proper noun sentences • 1.5h retainer engagement

Eastfield Tortoise TTS Ltd. built a high-quality audiobook production service using Tortoise TTS for voice cloning from a client narrator’s reference recordings. The ‘standard’ preset (N=512 CLVP candidates) produced excellent results — accurate pronunciation of everything including rare vocabulary and proper nouns — but required 45 seconds per sentence, making 10-hour audiobooks prohibitively slow to produce in batch. The ‘fast’ preset (N=4 candidates) reduced latency to 12 seconds per sentence, acceptable for batch production, and produced correct results for the vast majority of text. But sentences containing rare proper nouns — character surnames like “Featherstone” and “Ravenscroft”, place names like “Blantyre” and “Llandrindod” — were mispronounced in 20–30% of fast-preset outputs. The “th” in “Featherstone” became a /d/ sound (“Feaderstone”) in roughly one in five outputs. “Ravenscroft” received first-syllable stress in roughly three in ten outputs (“RAvenscroft” instead of “raVENscroft”).

The team tried different voice conditioning clips (different reference audio segments, different clip counts from 1 to 6), tried different random seeds, and consulted the Tortoise TTS documentation on the CLVP reranking step. They found that increasing the random seed diversity had no effect on the mispronunciation pattern — the same names were always mispronounced in the same ways, suggesting a systematic model preference rather than stochastic variation. After three weeks without resolution, they engaged a Tortoise TTS retainer developer who identified that the CLVP reranking pool of N=4 fast-preset candidates was too small to include the correct pronunciation for rare proper nouns, because the autoregressive model assigned high probability to incorrect pronunciations, and all four generated candidates reflected that same high-probability error.

The Tortoise TTS inference pipeline at the time of the engagement:

# Eastfield Tortoise TTS Ltd. — audiobook production pipeline
# Tortoise TTS (neonbjb/tortoisetesting), Python 3.11, PyTorch 2.1.x
# Use case: high-quality audiobook production with voice cloning

import torch
import time
from tortoise.api import TextToSpeech
from tortoise.utils.audio import load_voices

tts = TextToSpeech()

# Voice conditioning: 4 clips of narrator reference audio
voice_samples, conditioning_latents = load_voices(
    ['narrator_reference'],
    extra_voice_dirs=['voice_conditioning/']
)

def synthesize_audiobook_buggy(sentence, preset='fast'):
    """
    Synthesize a single audiobook sentence.

    Tortoise TTS architecture:
    1. Autoregressive (GPT-based) model generates N candidate text token sequences
       (quantized audio tokens representing the speech)
    2. CLVP (Contrastive Language-Voice Pretraining) scores each candidate:
       - Measures alignment between text and voice conditioning
       - Selects the best candidate based on CLVP score
    3. Diffusion model decodes selected candidate tokens → mel spectrogram
    4. Vocoder converts mel spectrogram → waveform

    'fast' preset: num_autoregressive_samples=4 (N=4 CLVP candidates)
    'standard' preset: num_autoregressive_samples=512 (N=512 CLVP candidates)

    Bug: for rare proper nouns, N=4 candidates all share the same mispronunciation.
    CLVP selects the best of four wrong answers.
    """
    start = time.perf_counter()

    gen = tts.tts_with_preset(
        text=sentence,
        voice_samples=voice_samples,
        conditioning_latents=conditioning_latents,
        preset=preset,  # 'fast' = N=4; 'standard' = N=512
    )

    latency_s = time.perf_counter() - start
    return gen, latency_s

# Mispronunciation analysis — fast preset (N=4), 100 trials per name:
#
# "Featherstone" (English surname — medial -th- consonant cluster):
#   Correct:  /fɛð.ə.stəʊn/ — 'th' is dental fricative /ð/
#   Error:    /fɛd.ə.stəʊn/ — 'th' produced as dental stop /d/ (20/100 trials)
#   Error pattern: stable — same 20 trials always produce /d/, same 80 always /ð/
#   Random seed effect: negligible — mispronunciation rate stable across seeds
#
# "Ravenscroft" (English surname — polysyllabic stress placement):
#   Correct:  /rəˈvɛnz.krɒft/ — stress on second syllable (raVENscroft)
#   Error:    /ˈreɪ.vənz.krɒft/ — stress on first syllable (RAvenscroft) (28/100)
#
# "Llandrindod" (Welsh place name — initial Ll- digraph):
#   Correct:  /ɬanˈdrɪn.dɒd/ — voiceless lateral fricative /ɬ/
#   Error:    /llanˈdrɪn.dɒd/ — regular /l/ instead of /ɬ/ (41/100)
#
# Root cause hypothesis: autoregressive model P(mispronunciation) is high enough
# that all N=4 fast-preset candidates draw the same error.
# P(error per sample) ≈ 0.20–0.41 for these names.
# P(all 4 candidates correct) = (1 - P(error))^4
#   "Featherstone": (0.80)^4 = 0.41 — 59% chance at least one is wrong; not the issue
#   Wait — if P(error) is the probability per sample, and standard (N=512) corrects it,
#   the issue is that P(error) is the DOMINANT mode, not a minority mode.
#
# Revised analysis: P(correct per sample) ≈ 0.03–0.08 for these names.
# The autoregressive model rarely generates the correct pronunciation.
# P(at least one correct in N candidates) = 1 - (1 - P_correct)^N
#   N=4:   1 - (0.95)^4  = 0.185 — only 18.5% chance of at least one correct sample
#          (matches observed: fast preset correct 80% of time for most proper nouns)
#   N=4:   1 - (0.97)^4  = 0.114 — 11.4% chance (Llandrindod: correct only 59%)
#   N=512: 1 - (0.95)^512 ≈ 1.000 — essentially guaranteed at least one correct
#
# CLVP selects the best from the N candidates.
# If 0/4 candidates are correct, CLVP selects the least-wrong wrong answer.
# If ≥1/512 candidates are correct, CLVP identifies and selects the correct one.

print("Root cause: P(correct pronunciation per autoregressive sample) ~3-8% for rare proper nouns")
print("N=4: P(≥1 correct in pool) ≈ 11-28% → CLVP often has no correct option")
print("N=512: P(≥1 correct in pool) ≈ 100% → CLVP always has a correct option to select")

Why CLVP reranking requires large N to correct rare proper noun pronunciation

Tortoise TTS’s quality advantage over non-reranked autoregressive TTS models comes entirely from CLVP: by generating a large pool of candidate waveforms and scoring each with the contrastive model, Tortoise TTS can select a candidate that matches the voice conditioning better than any single sample from the autoregressive model. For common words, the autoregressive model’s training data contains abundant examples of correct pronunciation, so the model’s distribution over pronunciation variants is dominated by the correct form. The top-4 fast-preset candidates are almost always correct, and CLVP selects among acceptable options. For rare proper nouns with irregular or non-English phoneme sequences, the model may never have seen the name during training, or may have seen it rarely enough that incorrect-but-similar phoneme patterns from common words dominate the probability distribution. For “Featherstone”, the medial consonant cluster “-atherst-” resembles patterns the model has seen with /d/ (from words like “fathered”, “gathered”, “lathered”) more often than patterns with /ð/ in the same context. If the model assigns P(correct /ð/) ≈ 0.05 per autoregressive sample, N=4 produces at least one correct sample only 18.5% of the time, leaving CLVP with four mispronounced candidates in 81.5% of fast-preset runs.

The fix is to detect sentences containing rare proper nouns before synthesis and apply a larger candidate count selectively to those sentences. For the audiobook domain, a spaCy NER tagger identifies PERSON, GPE (location), and ORG entities in each sentence. Sentences containing NER entities with tokens of 10 or more characters are marked as “rare proper noun sentences” and synthesized with N=32 CLVP candidates. For N=32, P(at least one correct) rises from 18.5% (N=4) to approximately 82% for a name with P(correct) = 0.05 per sample. The remaining 18% of cases where N=32 still fails to surface a correct candidate can be addressed by an optional pronunciation dictionary lookup for the specific book’s named entities, which overrides the autoregressive model’s output with a manually specified phoneme sequence for the most problematic names. All other sentences continue using N=4, preserving throughput for the 90%+ of sentences that contain no rare proper nouns.

# Fix: NER-based selective CLVP candidate count
# spaCy NER detects rare proper noun sentences; N=32 for those; N=4 for all others

import spacy
import time
from tortoise.api import TextToSpeech
from tortoise.utils.audio import load_voices

tts = TextToSpeech()
voice_samples, conditioning_latents = load_voices(
    ['narrator_reference'],
    extra_voice_dirs=['voice_conditioning/']
)

# Load spaCy NER model for entity detection
nlp = spacy.load('en_core_web_sm')

# Thresholds for rare proper noun classification
RARE_NOUN_MIN_CHARS = 10      # Minimum token length to classify as rare proper noun
RARE_NOUN_TYPES = {'PERSON', 'GPE', 'ORG', 'LOC'}  # NER entity types
RARE_NOUN_N = 32              # Candidate count for sentences with rare proper nouns
FAST_N = 4                    # Default fast candidate count

# Optional: book-specific pronunciation dictionary
# Maps {token: phoneme_string} for most problematic proper nouns
PRONUNCIATION_DICT = {
    'Llandrindod':  'ɬanˈdrɪn.dɒd',   # Welsh Ll- digraph
    'Ravenscroft':  'rəˈvɛnz.krɒft',   # second-syllable stress
    'Featherstone': 'fɛð.ə.stəʊn',     # medial /ð/
}

def classify_sentence(sentence):
    """
    Returns True if sentence contains a rare proper noun requiring N=32.

    A sentence is a rare proper noun sentence if it contains any NER entity
    of type PERSON, GPE, ORG, or LOC with a token of >= 10 characters.
    Short tokens (< 10 chars) are common enough that N=4 is sufficient.
    """
    doc = nlp(sentence)
    for ent in doc.ents:
        if ent.label_ not in RARE_NOUN_TYPES:
            continue
        for token in ent:
            if len(token.text) >= RARE_NOUN_MIN_CHARS:
                return True, ent.text
    return False, None

def synthesize_audiobook_fixed(sentence):
    """
    Synthesize audiobook sentence with selective CLVP candidate count.
    - Rare proper noun sentences: N=32 (surface low-P correct pronunciation)
    - All other sentences: N=4 (fast preset latency)
    """
    is_rare_noun, entity_text = classify_sentence(sentence)

    if is_rare_noun:
        # Apply larger candidate count for rare proper noun sentences
        num_candidates = RARE_NOUN_N
    else:
        num_candidates = FAST_N

    start = time.perf_counter()

    gen = tts.tts_with_preset(
        text=sentence,
        voice_samples=voice_samples,
        conditioning_latents=conditioning_latents,
        preset='custom',
        # Override candidate count based on NER classification
        num_autoregressive_samples=num_candidates,
        diffusion_iterations=200,   # Same quality diffusion for all
        cond_free=True,
        cond_free_k=2,
        temperature=0.8,
    )

    latency_s = time.perf_counter() - start
    return gen, latency_s, is_rare_noun

def synthesize_chapter(sentences):
    """
    Synthesize a chapter of audiobook text with selective preset switching.
    Logs per-sentence latency and proper noun classification.
    """
    results = []
    for i, sentence in enumerate(sentences):
        audio, latency, is_rare = synthesize_audiobook_fixed(sentence)
        results.append({
            'index': i,
            'sentence': sentence[:60] + ('...' if len(sentence) > 60 else ''),
            'is_rare_noun': is_rare,
            'candidates_used': RARE_NOUN_N if is_rare else FAST_N,
            'latency_s': latency,
        })
        print(f"  [{i:3d}] {'RARE ' if is_rare else '     '} "
              f"N={results[-1]['candidates_used']:3d}  "
              f"{latency:.1f}s  {results[-1]['sentence']}")

    # Summary statistics
    rare_count = sum(1 for r in results if r['is_rare_noun'])
    fast_count = len(results) - rare_count
    avg_latency_fast = (sum(r['latency_s'] for r in results if not r['is_rare_noun'])
                        / max(fast_count, 1))
    avg_latency_rare = (sum(r['latency_s'] for r in results if r['is_rare_noun'])
                        / max(rare_count, 1))
    print(f"\nChapter summary: {len(results)} sentences, "
          f"{rare_count} rare noun ({rare_count/len(results):.0%}), "
          f"{fast_count} fast")
    print(f"Average latency: fast {avg_latency_fast:.1f}s, rare {avg_latency_rare:.1f}s")

    return results

# ---- Results after NER-based selective candidate count ----
#
# Mispronunciation rate (100 trials per name):
#   "Featherstone" (th→/ð/):
#     Before fix (N=4):   20/100 trials mispronounced (20% error rate)
#     After fix (N=32):    2/100 trials mispronounced (2% error rate)
#
#   "Ravenscroft" (second-syllable stress):
#     Before fix (N=4):   28/100 trials mispronounced (28% error rate)
#     After fix (N=32):    3/100 trials mispronounced (3% error rate)
#
#   "Llandrindod" (Welsh Ll-):
#     Before fix (N=4):   41/100 trials mispronounced (41% error rate)
#     After fix (N=32):   18/100 trials mispronounced (18% error rate)
#     Note: Welsh Ll- remains challenging; pronunciation dictionary recommended
#
# Per-sentence latency (A100 GPU):
#   Common text, N=4 (fast preset):        12.1 s mean (unchanged)
#   Rare proper noun sentence, N=32:        28.3 s mean (+16.2 s vs fast)
#   Chapter mix (8% rare noun sentences):   13.4 s mean per sentence
#   vs all-standard N=512:                  45.0 s mean per sentence
#   Throughput improvement vs standard:     3.4× faster for mixed chapters
#
# NER classification accuracy (200-sentence audiobook sample):
#   True positive (rare noun sentence correctly classified): 94%
#   False negative (rare noun sentence missed, used N=4):    6%
#   False positive (common sentence misclassified as rare):  3%
#   (False positives cost 16s extra latency; acceptable)

print("Fix: NER selective N=32; mispronunciation 20% → 2% for most rare proper nouns")
print("Chapter throughput: 13.4s mean vs 45s all-standard — 3.4× faster")

Related retainer posts

• Bark developer on retainer — autoregressive TTS; long-sequence semantic token repetition loop in outputs longer than ~15 seconds; fix: repetition penalty and temperature scheduling during autoregressive decoding — 3 weeks
• FastSpeech 2 developer on retainer — non-autoregressive TTS with duration, pitch, energy predictors; duration predictor systematic short duration on proper noun consonant clusters; fix: duration prediction fine-tuning on proper noun corpus — 3 weeks
• VITS developer on retainer — 2-hour YouTuber dataset; 15–20% utterance-final vocal fry; fine-tuning learns creaky voice from YouTuber speech; fix: trim last 100–200 ms where F0 < 80 Hz; re-fine-tune — 3 weeks