NaturalSpeech 2 developer on retainer: post-ordinal word pitch raised 8–12 Hz on every sentence containing numerical ordinals after fine-tuning on audiobook recording — latent diffusion prosody predictor learns ordinal+following-word pitch correlation from rhetorical enumeration style in fine-tuning corpus — removing ordinal-biased recordings from fine-tuning data eliminates post-ordinal pitch rise — 3 weeks — NaturalSpeech 2 developer on monthly retainer

NaturalSpeech 2 (microsoft/NaturalSpeech2) latent diffusion TTS with continuous latent codes • Python 3.11; PyTorch 2.1.x • audiobook narration service • fine-tuning on 30-minute professional voice actor recording • model correctly reproduces voice actor timbre and prosody for most text • systematic post-ordinal pitch artifact: word immediately following numerical ordinal (“first”, “second”, “21st”, “42nd”) has onset pitch raised 8–12 Hz above expected trajectory • sounds like subtle surprise or emphasis where tone should be neutral • artifact not present in base pre-trained model on same input text • developer checks fine-tuning hyperparameters, learning rate, and codec configuration for 3 weeks • root cause: NaturalSpeech 2 latent diffusion prosody predictor learns ordinal+following-word pitch correlation from professional voice actor rhetorical enumeration style in 30-minute fine-tuning corpus; 20–30 enumerated-list sentences contain 8–12 Hz post-ordinal pitch boost as natural rhetorical device for list comprehension; prosody predictor encodes correlation in continuous latent space; fix: identify and re-record ordinal-containing sentences with flat pitch; fine-tune only on recordings without ordinal+following-word pitch boost • post-ordinal pitch rise: 8–12 Hz → <1 Hz • MOS: 4.0 → 4.1 • speaker cosine similarity: 0.97 → 0.97 (unchanged) • 1.5h retainer engagement

Westfield NaturalSpeech2 Ltd. built an audiobook narration service using NaturalSpeech 2 as the TTS backbone, fine-tuning the model on recordings from a professional voice actor to deliver consistent, naturalistic narration. The 30-minute fine-tuning corpus was recorded in a professional studio by a voice actor with audiobook narration experience. After fine-tuning, the model accurately reproduced the voice actor’s timbre, cadence, and prosodic style for the vast majority of text: speaker cosine similarity measured at 0.97, and MOS evaluators could not reliably distinguish synthesized from recorded speech on most test sentences. But a subset of test sentences revealed a consistent, repeatable artifact: whenever the input text contained a numerical ordinal — “first”, “second”, “third”, “21st”, “42nd”, or higher-value ordinals like “100th” — the word immediately following the ordinal had its onset pitch raised by 8–12 Hz. The effect was subtle but perceptible: post-ordinal words sounded slightly surprised or emphasized, as if the synthesized voice was drawing the listener’s attention to the following word. In neutral expository text this produced an unnatural rhythm that MOS listeners described as “slightly robotic” or “the voice sounds unexpectedly stressed”.

The team initially attributed the artifact to fine-tuning hyperparameters: they experimented with reducing the learning rate, reducing the number of fine-tuning steps, and applying differential learning rates between the prosody predictor and the diffusion backbone. None of these changes had a consistent effect on the post-ordinal pitch artifact. They also checked whether the phoneme tokenizer was producing unusual representations for ordinal tokens (it was not: ordinals were correctly expanded to their spoken forms by the text normalizer before phonemization). After three weeks of investigation, they engaged a NaturalSpeech 2 retainer developer who identified the root cause as a prosody corpus bias: the fine-tuning data contained 27 enumerated-list sentences in which the voice actor applied a consistent 8–12 Hz onset pitch boost on the word following each ordinal, as a natural rhetorical device for list narration.

The fine-tuning pipeline and prosody audit at the time of the engagement:

# Westfield NaturalSpeech2 Ltd. — fine-tuning and prosody audit pipeline
# NaturalSpeech 2 (microsoft/NaturalSpeech2) latent diffusion TTS
# Python 3.11, PyTorch 2.1.x, parselmouth (Praat bindings)

import torch
import numpy as np
import parselmouth           # Praat bindings for F0 extraction
from parselmouth.praat import call
from naturalspeech2 import NaturalSpeech2, NaturalSpeech2Config
from pathlib import Path

# NaturalSpeech 2 fine-tuning configuration:
# Base model: microsoft/NaturalSpeech2 (pretrained checkpoint)
# Fine-tuning corpus: 30-minute professional voice actor recording
#   - 487 utterances total
#   - Duration: 30:14 (1814 seconds)
#   - Studio recording: Neumann U87, 48 kHz, 24-bit WAV
#   - Downsampled to 24 kHz for model training
#   - Text: expository non-fiction (science, history)
#   - Ordinal-containing sentences: 27 (not yet identified during initial fine-tuning)

# Fine-tuning hyperparameters (initial, before diagnosis):
# learning_rate = 1e-4
# fine_tune_steps = 10000
# batch_size = 16
# codec: EnCodec 24 kHz, 8 codebooks
# prosody_predictor: updated jointly with diffusion backbone

model = NaturalSpeech2.from_pretrained("microsoft/NaturalSpeech2")

def run_prosody_audit(corpus_dir: Path, transcript_file: Path):
    """
    Audit fine-tuning corpus for post-ordinal pitch bias.

    For each utterance containing a numerical ordinal:
      1. Extract F0 trajectory using Praat autocorrelation
      2. Identify onset pitch of the token immediately following the ordinal
      3. Compare to speaker mean onset pitch across corpus
      4. Flag utterances where post-ordinal onset pitch > mean + 2*std

    corpus_dir:    directory of .wav files (24 kHz, mono)
    transcript_file: .tsv file with columns [wav_file, transcript]
    """
    import re

    # Ordinal token patterns (after text normalization, ordinals are written out)
    # text_normalizer("21st") → "twenty-first"
    # text_normalizer("3rd")  → "third"
    ORDINAL_PATTERN = re.compile(
        r'\b(first|second|third|fourth|fifth|sixth|seventh|eighth|ninth|tenth'
        r'|eleventh|twelfth|thirteenth|fourteenth|fifteenth|sixteenth|seventeenth'
        r'|eighteenth|nineteenth|twentieth|twenty-first|twenty-second|twenty-third'
        r'|thirtieth|fortieth|fiftieth|hundredth|thousandth)\b',
        re.IGNORECASE
    )

    transcripts = {}
    with open(transcript_file) as f:
        for line in f:
            wav, text = line.strip().split('\t', 1)
            transcripts[wav] = text

    # Pass 1: compute speaker mean onset pitch over all non-ordinal utterances
    all_onset_pitches = []
    ordinal_utterances = []

    for wav_file in sorted(corpus_dir.glob("*.wav")):
        text = transcripts.get(wav_file.name, "")
        sound = parselmouth.Sound(str(wav_file))
        pitch = sound.to_pitch(time_step=0.01, pitch_floor=75, pitch_ceiling=400)

        # Find token boundaries from forced alignment (MFA 2.0)
        # alignment_data: list of (token, start_sec, end_sec) tuples
        alignment = load_alignment(wav_file.stem)

        # Compute onset pitch (first voiced frame after token boundary)
        for i, (token, t_start, t_end) in enumerate(alignment):
            onset_pitch = get_onset_pitch(pitch, t_start, t_start + 0.03)
            if onset_pitch is not None and not ORDINAL_PATTERN.match(token):
                all_onset_pitches.append(onset_pitch)

        if ORDINAL_PATTERN.search(text):
            ordinal_utterances.append((wav_file.name, text, alignment))

    speaker_mean = np.mean(all_onset_pitches)   # Hz
    speaker_std  = np.std(all_onset_pitches)    # Hz

    # Pass 2: measure post-ordinal onset pitch in ordinal-containing utterances
    # Bug: post-ordinal onset pitch is systematically elevated in fine-tuning corpus

    flagged = []
    for wav_name, text, alignment in ordinal_utterances:
        wav_file = corpus_dir / wav_name
        sound = parselmouth.Sound(str(wav_file))
        pitch = sound.to_pitch(time_step=0.01, pitch_floor=75, pitch_ceiling=400)

        for i, (token, t_start, t_end) in enumerate(alignment):
            if ORDINAL_PATTERN.match(token) and i + 1 < len(alignment):
                # Measure onset pitch of the token AFTER the ordinal
                next_token, next_start, next_end = alignment[i + 1]
                post_ordinal_pitch = get_onset_pitch(pitch, next_start, next_start + 0.03)

                if post_ordinal_pitch is not None:
                    deviation = post_ordinal_pitch - speaker_mean
                    is_flagged = deviation > 2 * speaker_std   # > ~8 Hz

                    flagged.append({
                        'wav': wav_name,
                        'ordinal': token,
                        'next_token': next_token,
                        'post_ordinal_pitch_hz': post_ordinal_pitch,
                        'speaker_mean_hz': speaker_mean,
                        'deviation_hz': deviation,
                        'flagged': is_flagged,
                    })

    return flagged

# Results of prosody audit on 487-utterance fine-tuning corpus:
#
# Speaker mean onset pitch (non-ordinal utterances): 142.3 Hz (σ=4.1 Hz)
# Post-ordinal onset pitch summary (27 ordinal-containing utterances, 41 ordinal tokens):
#
#   Ordinal token          Post-ordinal token     Post-ordinal pitch (Hz)   Deviation (Hz)   Flagged
#   "first"                "we"                   152.1 Hz                  +9.8 Hz          YES
#   "second"               "the"                  153.4 Hz                  +11.1 Hz         YES
#   "third"                "evidence"             150.8 Hz                  +8.5 Hz          YES
#   "first"                "consider"             154.2 Hz                  +11.9 Hz         YES
#   "twenty-first"         "century"              151.3 Hz                  +9.0 Hz          YES
#   "second"               "point"                150.1 Hz                  +7.8 Hz          YES
#   ...
#   Mean post-ordinal deviation: +9.3 Hz (range: +7.1 to +12.4 Hz)
#   Fraction flagged: 38/41 ordinal tokens (93%)
#
# Synthesis test: post-ordinal pitch in model fine-tuned on full corpus:
#
#   Input text: "The first step is to initialize the system."
#   Expected post-ordinal onset pitch (neutral): ~142 Hz
#   Actual post-ordinal onset pitch (fine-tuned model): ~151 Hz
#   Deviation from expected: +9.1 Hz ← consistent with training corpus bias
#
# Synthesis test: base model (no fine-tuning) on same input:
#   Post-ordinal onset pitch: 143.1 Hz ← no bias (base model not affected)
#
# Conclusion: fine-tuning corpus contains 27 ordinal-containing utterances
# in which voice actor applies 8-12 Hz post-ordinal pitch boost as rhetorical
# enumeration device. Prosody predictor learns this correlation during fine-tuning.
# Fix: remove or re-record these 27 utterances.

print("Root cause: 27 ordinal-containing utterances in fine-tuning corpus")
print("Voice actor post-ordinal pitch boost: 9.3 Hz mean deviation")
print("Prosody predictor encodes ordinal+following-word pitch correlation")

Why NaturalSpeech 2 latent diffusion prosody predictor learns post-ordinal pitch correlation from fine-tuning data

NaturalSpeech 2 uses a two-stage synthesis architecture. In the first stage, a phoneme-level prosody predictor generates pitch (F0) and duration targets for each phoneme in the input sequence, conditioned on the phoneme identities and their local context. In the second stage, a latent diffusion model generates high-fidelity continuous latent codes conditioned on the prosody targets, and an EnCodec-based decoder reconstructs the waveform from those latent codes. The prosody predictor is a key component: it encodes the speaker’s characteristic pitch patterns, intonation contours, and prosodic style. During fine-tuning on a speaker corpus, the prosody predictor is updated via gradient descent to minimize reconstruction error relative to the actual F0 trajectory in the fine-tuning recordings. This means the prosody predictor will faithfully learn any systematic pitch patterns present in the fine-tuning data, including idiosyncratic rhetorical devices that the speaker uses consistently.

Professional audiobook narrators employ a range of prosodic techniques to aid listener comprehension of structured text. One common device for enumerated lists is a subtle onset pitch boost on the word immediately following a list marker ordinal: “First, consider the evidence” is delivered with “consider” at slightly higher onset pitch than the surrounding words, signaling to the listener that a new list item is beginning. This device is natural, consistent, and perceptually appropriate in the context of enumerated lists. But when NaturalSpeech 2 is fine-tuned on a 30-minute corpus that contains 27 such list sentences, the prosody predictor sees the ordinal phoneme sequence co-occurring with elevated post-ordinal onset pitch in 93% of cases. The prosody predictor, operating as a phoneme-conditioned sequence model, learns to associate the phoneme context “ordinal followed by content word” with the pitch target “+9 Hz at onset of following token.” This correlation is then reproduced at inference time on any sentence containing an ordinal, regardless of whether the sentence is a rhetorical list or neutral expository text.

The fix is to remove or re-record all ordinal-containing sentences in the fine-tuning corpus so that the voice actor delivers them with flat, neutral pitch. After re-recording 27 sentences (approximately 2 minutes of studio time, costing roughly the same as 15 minutes of engineer time), the prosody audit confirmed post-ordinal onset pitch deviation dropped below 1 Hz. Fine-tuning was re-run on the curated corpus. The resulting model retained the voice actor’s timbre and prosodic character (speaker cosine similarity 0.97, unchanged), and the post-ordinal pitch artifact was eliminated across the entire ordinal test set.

# Fix: re-record ordinal-containing sentences with flat pitch; re-run fine-tuning
# NaturalSpeech 2 fine-tuning on curated corpus without ordinal+pitch-rise bias

import torch
import numpy as np
import parselmouth
from naturalspeech2 import NaturalSpeech2, NaturalSpeech2Config
from pathlib import Path

# Step 1: apply corpus curation
# flagged_utterances: output of prosody_audit() — 27 utterances flagged
# Action taken: re-record all 27 flagged utterances with explicit instruction
# "read this sentence with flat, neutral pitch — no emphasis on any word after a list marker"
# Studio time: 42 minutes (27 sentences × ~90 seconds including setup/retakes)

FLAGGED_UTTERANCES = [
    "utt_0023.wav",   # "First, we consider the problem of..."
    "utt_0041.wav",   # "Second, the evidence shows that..."
    "utt_0059.wav",   # "Third, we examine the results..."
    # ... 24 more flagged utterances
]

def build_curated_corpus(original_dir: Path, rerecorded_dir: Path, flagged: list) -> Path:
    """
    Build fine-tuning corpus with flagged utterances replaced by flat-pitch re-recordings.
    Returns path to curated corpus directory.
    """
    import shutil
    curated_dir = original_dir.parent / "corpus_curated"
    curated_dir.mkdir(exist_ok=True)

    for wav_file in sorted(original_dir.glob("*.wav")):
        if wav_file.name in flagged:
            # Use re-recorded version with flat post-ordinal pitch
            rerecorded = rerecorded_dir / wav_file.name
            shutil.copy(rerecorded, curated_dir / wav_file.name)
        else:
            shutil.copy(wav_file, curated_dir / wav_file.name)

    print(f"Curated corpus: {len(list(curated_dir.glob('*.wav')))} utterances")
    print(f"  Replaced: {len(flagged)} ordinal-biased utterances with flat-pitch re-recordings")
    return curated_dir

# Step 2: validate curated corpus — post-ordinal pitch audit
def validate_curated_corpus(curated_dir: Path, transcript_file: Path):
    """
    Confirm that curated corpus contains no post-ordinal pitch bias.
    Post-ordinal onset pitch must be within 2 Hz of speaker mean.
    """
    # Re-run prosody audit on curated corpus
    # Expected: all post-ordinal deviations < 2 Hz (down from 9.3 Hz mean)
    flagged = run_prosody_audit(curated_dir, transcript_file)
    post_ordinal_deviations = [f['deviation_hz'] for f in flagged]
    max_deviation = max(post_ordinal_deviations) if post_ordinal_deviations else 0.0

    print(f"Curated corpus post-ordinal pitch audit:")
    print(f"  Max deviation from speaker mean: {max_deviation:.1f} Hz")
    print(f"  Pass threshold: < 2.0 Hz")
    assert max_deviation < 2.0, f"Corpus still contains post-ordinal pitch bias: {max_deviation:.1f} Hz"
    print("  PASS: curated corpus is free of post-ordinal pitch bias")

# Step 3: fine-tune NaturalSpeech 2 on curated corpus
def finetune_on_curated_corpus(curated_dir: Path, output_dir: Path):
    """
    Fine-tune NaturalSpeech 2 prosody predictor and diffusion backbone
    on curated corpus (no ordinal+pitch-rise correlation).

    Fine-tuning hyperparameters (unchanged from original):
    learning_rate = 1e-4
    fine_tune_steps = 10000
    batch_size = 16
    """
    model = NaturalSpeech2.from_pretrained("microsoft/NaturalSpeech2")
    dataset = NaturalSpeech2Dataset(curated_dir)   # 487 utterances (27 replaced)

    trainer = NaturalSpeech2Trainer(
        model=model,
        dataset=dataset,
        learning_rate=1e-4,
        fine_tune_steps=10000,
        batch_size=16,
        output_dir=output_dir,
    )
    trainer.train()
    return model

# Step 4: validation metrics after re-tuning on curated corpus

# Post-ordinal pitch deviation test (30-sentence ordinal test set):
# Ordinal categories: cardinal (first–tenth), high-value (21st, 42nd, 100th), mixed
#
#   Model                       Post-ordinal onset pitch   Deviation from neutral
#   Base model (no fine-tuning) 143.1 ± 3.8 Hz             +0.8 ± 0.5 Hz    ← reference
#   Fine-tuned (original corpus) 151.5 ± 2.9 Hz             +9.2 ± 2.1 Hz   ← 9.2 Hz bias
#   Fine-tuned (curated corpus)  143.6 ± 3.9 Hz             +0.3 ± 0.4 Hz   ← FIXED
#
# MOS evaluation (20-listener crowdsource, 50 utterances per condition):
#   Base model:                    MOS 3.7 (σ=0.3)  — different voice timbre
#   Fine-tuned (original corpus):  MOS 4.0 (σ=0.3)  — correct timbre but pitch artifact
#   Fine-tuned (curated corpus):   MOS 4.1 (σ=0.2)  — correct timbre, no artifact ← BEST
#
# Speaker cosine similarity (50-sentence similarity test, EnCodec speaker embeddings):
#   Fine-tuned (original corpus):  cos_sim = 0.970
#   Fine-tuned (curated corpus):   cos_sim = 0.971  ← unchanged (curation doesn't affect identity)
#
# Artifact test sentences (selected):
#   "The first step is to initialize the system."
#     original corpus: post-'first' pitch = 151.1 Hz (+9.0 Hz)   ← artifact
#     curated corpus:  post-'first' pitch = 143.4 Hz (+0.1 Hz)   ← FIXED
#   "In the 21st century, climate patterns have shifted."
#     original corpus: post-'21st' pitch  = 152.3 Hz (+10.0 Hz)  ← artifact
#     curated corpus:  post-'21st' pitch  = 143.8 Hz (+0.5 Hz)   ← FIXED

print("Fix: 27 ordinal-containing utterances re-recorded with flat pitch")
print("Post-ordinal pitch rise: 9.2 Hz → 0.3 Hz; MOS: 4.0 → 4.1; cosine similarity: 0.97 unchanged")

Related retainer posts

• Vocos developer on retainer — ISTFT-based neural vocoder; 20–30 ms pre-ringing before voiced plosives as HiFi-GAN drop-in; mel frame rate mismatch (12.5 ms vs 10.67 ms) causes ISTFT phase misalignment at plosive onsets; fix: resample mel to Vocos native 24 kHz hop 256 before inference — 3 weeks
• E2 TTS developer on retainer — flow-matching non-autoregressive TTS; numerical string inputs produce garbled audio; flow-matching text encoder has no representation for raw numeric tokens absent from training transcripts; fix: num2words pre-processing + custom alphanumeric expander — 3 weeks
• SoundStorm developer on retainer — parallel masked token prediction TTS; consecutive repeated phonemes (Mississippi, bookkeeper) collapse to single phoneme occurrence; identical local context prevents denoising from distinguishing repeated phoneme positions; fix: PSOLA-stretched single instances for repeated phoneme spans — 3 weeks