AdaSpeech developer on retainer: monotone speaker adaptation produces 4–5 Hz pitch wobble artifact not present in target speaker’s voice — variance adapter not updated during speaker adaptation retains base model’s learned 4–5 Hz pitch variance cycle — including variance adapter normalization parameters in adaptation parameter set eliminates wobble — 3 weeks — AdaSpeech developer on monthly retainer

AdaSpeech (microsoft/AdaSpeech) adaptive TTS with speaker adaptation • Python 3.11; PyTorch 2.1.x • personalized voice assistant • correct speech when adapted to speakers with moderate pitch variability, but monotone speaker adaptation (pitch coefficient of variation < 0.1) produces periodic 4–5 Hz vibrato-like wobble artifact not present in target speaker’s voice • high-variability speaker adaptation correct • developer checks AdaSpeech pitch predictor, acoustic adapter, and variance adapter for 3 weeks • root cause: AdaSpeech acoustic adapter updates only layer normalization parameters; variance adapter not updated during adaptation; base model variance adapter trained on moderate-variability speakers learned 4–5 Hz pitch variance cycle as default pitch dynamics; for monotone target, variance adapter oscillation not suppressed because it was not updated • fix: include variance adapter speaker-specific normalization parameters in adaptation parameter set; recalibrate using zero-variance pitch targets during fine-tuning • wobble rate: 95% of monotone-speaker utterances → 5% • MOS: 3.5 → 3.9 • 1.5h retainer engagement

Eastfield AdaSpeech Ltd. deployed AdaSpeech for personalized voice synthesis in a voice assistant product. The system allowed users to record a short set of adaptation utterances and receive a synthesized voice that matched their own vocal characteristics. For the majority of users — people with normal to expressive pitch variation in their speech — AdaSpeech produced synthesis that listeners correctly identified as matching the target speaker, with natural-sounding prosody. But a subset of users reported that the synthesized version of their voice had an artificial, wavering quality — a subtle periodic pitch oscillation that was not present in their natural speech. These users were typically speakers with low pitch variability: calm, measured speakers, readers of technical content, and people whose natural speaking style was monotone or near-monotone.

Listeners described the artifact as a “vibrato” or “wobble” — a periodic pitch variation at approximately 4 to 5 Hz that made the synthesized voice sound as if the speaker were singing lightly rather than speaking. The wobble was not present in recordings of the target speakers’ natural voices, nor in the adaptation utterances they had recorded. It appeared only in synthesized output from the adapted model. Standard-variability speaker adaptations showed no such artifact, and the base (unadapted) model also synthesized without the wobble. The artifact was introduced by the interaction between AdaSpeech’s adaptation mechanism and the low-variability speaker profile.

The team initially checked whether the adaptation data quality was at fault — that the monotone speakers’ recordings had insufficient prosodic range to anchor AdaSpeech’s pitch predictor. They tried collecting additional adaptation utterances with more varied prosody from the same speakers. The wobble persisted. They then investigated whether the acoustic model was applying a prosody injection from an incorrect speaker embedding. After three weeks, they engaged an AdaSpeech retainer developer who traced the root cause to the variance adapter not being included in the adaptation parameter set, allowing its learned pitch oscillation to dominate the adapted voice.

The AdaSpeech adaptation configuration and pitch wobble diagnostic at the time of the engagement:

# Eastfield AdaSpeech Ltd. — AdaSpeech 4-5 Hz pitch wobble on monotone speaker adaptation
# AdaSpeech (microsoft/AdaSpeech) adaptive TTS with speaker adaptation
# Python 3.11, PyTorch 2.1.x
# Deployment: personalized voice assistant

import torch
import numpy as np
from adaspeech import AdaSpeech  # microsoft/AdaSpeech
from adaspeech.adaptation import adapt_to_speaker

SAMPLE_RATE = 22050
HOP_LENGTH = 256

# --- DIAGNOSTIC: measure F0 coefficient of variation for target speakers ---
# and detect 4-5 Hz pitch oscillation in synthesized speech

import librosa
import scipy.signal


def measure_f0_coefficient_of_variation(audio: np.ndarray, sr: int = SAMPLE_RATE) -> float:
    """
    Measure the F0 (pitch) coefficient of variation of a speech recording.
    CV = std(F0_voiced) / mean(F0_voiced) on voiced frames.
    CV < 0.10 indicates a monotone or near-monotone speaker.
    """
    f0, voiced_flag, _ = librosa.pyin(
        audio, fmin=librosa.note_to_hz('C2'),
        fmax=librosa.note_to_hz('C6'),
        sr=sr, frame_length=1024, hop_length=256
    )
    voiced_f0 = f0[voiced_flag]
    if len(voiced_f0) < 10:
        return 0.0
    return float(np.std(voiced_f0) / np.mean(voiced_f0))


def detect_f0_wobble(audio: np.ndarray, sr: int = SAMPLE_RATE) -> dict:
    """
    Detect 4-5 Hz periodic oscillation in synthesized speech F0 trajectory.
    Returns wobble frequency (Hz), amplitude (cents), and presence flag.
    """
    f0, voiced_flag, _ = librosa.pyin(
        audio, fmin=librosa.note_to_hz('C2'),
        fmax=librosa.note_to_hz('C6'),
        sr=sr, frame_length=1024, hop_length=256
    )
    voiced_f0 = f0[voiced_flag]
    if len(voiced_f0) < 50:
        return {'wobble_present': False, 'frequency_hz': 0.0, 'amplitude_cents': 0.0}

    # Convert F0 to cents (relative to mean)
    f0_cents = 1200 * np.log2(voiced_f0 / np.mean(voiced_f0))

    # Compute FFT of cents trajectory to detect periodic modulation
    # F0 frame rate: sr / hop_length = 22050 / 256 ≈ 86 Hz
    f0_frame_rate = sr / HOP_LENGTH  # ~86 Hz
    fft = np.abs(np.fft.rfft(f0_cents - f0_cents.mean()))
    freqs = np.fft.rfftfreq(len(f0_cents), d=1.0 / f0_frame_rate)

    # Find peak in 3-7 Hz range (wobble detection band)
    wobble_band = (freqs >= 3.0) & (freqs <= 7.0)
    if not wobble_band.any():
        return {'wobble_present': False, 'frequency_hz': 0.0, 'amplitude_cents': 0.0}

    band_amplitudes = fft[wobble_band]
    peak_idx = np.argmax(band_amplitudes)
    peak_freq = freqs[wobble_band][peak_idx]
    peak_amplitude = band_amplitudes[peak_idx] / len(f0_cents) * 2  # normalize

    return {
        'wobble_present': peak_amplitude > 3.0,  # > 3 cents amplitude is perceptible
        'frequency_hz': float(peak_freq),
        'amplitude_cents': float(peak_amplitude)
    }


# Load base model
model = AdaSpeech.from_pretrained('adaspeech_base')
model.eval()

# Speaker profiles
speakers = [
    {
        'id': 'speaker_A',
        'description': 'monotone speaker (CV < 0.1)',
        'adaptation_audio': load_adaptation_audio('speaker_A_adaptation.wav')
    },
    {
        'id': 'speaker_B',
        'description': 'standard speaker (CV ~0.25)',
        'adaptation_audio': load_adaptation_audio('speaker_B_adaptation.wav')
    }
]

# Standard AdaSpeech adaptation: acoustic layer norm parameters only
STANDARD_ADAPTATION_PARAMS = ['acoustic_decoder.layer_norm']
# Extended adaptation: acoustic layer norm + variance adapter layer norm
EXTENDED_ADAPTATION_PARAMS = [
    'acoustic_decoder.layer_norm',
    'variance_adapter.pitch_predictor.layer_norm',
    'variance_adapter.energy_predictor.layer_norm',
]

test_text = "The document has been prepared and is ready for your review."

for speaker in speakers:
    natural_cv = measure_f0_coefficient_of_variation(speaker['adaptation_audio'])
    print(f"\nSpeaker {speaker['id']} ({speaker['description']}, CV={natural_cv:.3f})")

    # Adapt with standard parameter set
    adapted_standard = adapt_to_speaker(
        model, speaker['adaptation_audio'],
        adaptation_params=STANDARD_ADAPTATION_PARAMS,
        n_steps=100
    )
    audio_standard = adapted_standard.synthesize(test_text)
    wobble_standard = detect_f0_wobble(audio_standard)

    print(f"  Standard adaptation (acoustic LN only):")
    print(f"    Wobble present: {wobble_standard['wobble_present']}")
    print(f"    Wobble frequency: {wobble_standard['frequency_hz']:.1f} Hz")
    print(f"    Wobble amplitude: {wobble_standard['amplitude_cents']:.1f} cents")

# Output:
# Speaker speaker_A (monotone speaker (CV < 0.1), CV=0.067)
#   Standard adaptation (acoustic LN only):
#     Wobble present: True
#     Wobble frequency: 4.3 Hz
#     Wobble amplitude: 7.8 cents   ← audible vibrato, clearly perceptible
#
# Speaker speaker_B (standard speaker (CV ~0.25), CV=0.241)
#   Standard adaptation (acoustic LN only):
#     Wobble present: False
#     Wobble frequency: 4.5 Hz
#     Wobble amplitude: 1.2 cents   ← below perceptual threshold, masked by natural variation
#
# Root cause confirmed: variance adapter oscillation (4-5 Hz) always present,
# but only perceptible in monotone speakers where natural F0 variation is below
# the oscillation amplitude

Why AdaSpeech’s variance adapter oscillation is audible only on monotone speakers

AdaSpeech’s speaker adaptation is designed around a minimal parameter set: only the layer normalization scale and bias parameters in the acoustic (mel spectrogram) decoder are updated during adaptation. This design choice is deliberate — it enables data-efficient adaptation with 20 utterances while preventing overfitting on the small adaptation corpus. The layer normalization parameters are sufficient to shift the acoustic decoder toward the target speaker’s mel spectrogram statistics, capturing voice quality, formant frequencies, and speaking rate.

What is not updated during standard AdaSpeech adaptation is the variance adapter — the module that predicts per-phoneme pitch (F0) and energy variance trajectories. The variance adapter is a FastSpeech 2-style predictor that takes phoneme embeddings as input and outputs pitch and energy sequences that modulate the acoustic decoder. During training on the multi-speaker base corpus, the variance adapter learns statistical patterns of pitch and energy dynamics across many speakers. One such learned pattern is a 4 to 5 Hz oscillation in the pitch trajectory during sustained voiced segments, which reflects the natural micro-variation in F0 that characterizes conversational speech with moderate expressiveness.

For speakers with moderate to high pitch variability, this learned oscillation falls within the natural F0 variation range. The synthesized F0 oscillates at 4 to 5 Hz with an amplitude of 1 to 3 cents — small enough to be perceived as natural micro-variation rather than an artifact. For a monotone speaker whose natural speech has F0 variation below 3 cents amplitude, the variance adapter’s 7 to 10 cent oscillation is markedly incongruent with the target speaker’s style. Because the variance adapter was frozen during adaptation, it has no mechanism to learn the monotone speaker’s near-flat F0 trajectory. The residual oscillation, unmasked by natural variation, manifests as a periodic wobble artifact.

# Fix: include variance adapter layer normalization parameters in adaptation parameter set
# For monotone speakers (CV < 0.15), extend adaptation to include variance adapter LN
# and recalibrate pitch targets toward zero-variance trajectory during fine-tuning

import torch
import numpy as np

MONOTONE_CV_THRESHOLD = 0.15  # speakers with CV below this get extended adaptation
EXTENDED_ADAPTATION_STEPS = 200  # more steps needed for variance adapter convergence


def select_adaptation_params(natural_cv: float) -> list:
    """
    Select adaptation parameter set based on target speaker F0 coefficient of variation.
    For monotone speakers (CV < 0.15), include variance adapter layer norm parameters.
    """
    base_params = ['acoustic_decoder.layer_norm']
    if natural_cv < MONOTONE_CV_THRESHOLD:
        variance_params = [
            'variance_adapter.pitch_predictor.layer_norm',
            'variance_adapter.energy_predictor.layer_norm',
        ]
        return base_params + variance_params
    return base_params


def build_zero_variance_pitch_targets(
    adaptation_audio: np.ndarray,
    sr: int = SAMPLE_RATE
) -> dict:
    """
    Extract mean F0 from adaptation audio for use as flat pitch targets
    during variance adapter recalibration.
    For monotone speakers, the target pitch trajectory is the mean F0
    with near-zero variance, expressed as small-amplitude noise around the mean.
    """
    f0, voiced_flag, _ = librosa.pyin(
        adaptation_audio,
        fmin=librosa.note_to_hz('C2'),
        fmax=librosa.note_to_hz('C6'),
        sr=sr, frame_length=1024, hop_length=256
    )
    voiced_f0 = f0[voiced_flag]
    mean_f0 = float(np.mean(voiced_f0)) if len(voiced_f0) > 0 else 120.0
    # Target: mean F0 with ±1 cent noise (essentially flat)
    return {'mean_f0': mean_f0, 'target_std_cents': 1.0}


def adapt_speaker_extended(
    model,
    adaptation_audio: np.ndarray,
    natural_cv: float,
) -> object:
    """
    Adapt AdaSpeech to speaker with monotone-aware parameter selection.
    """
    adaptation_params = select_adaptation_params(natural_cv)
    n_steps = EXTENDED_ADAPTATION_STEPS if natural_cv < MONOTONE_CV_THRESHOLD else 100

    if natural_cv < MONOTONE_CV_THRESHOLD:
        pitch_targets = build_zero_variance_pitch_targets(adaptation_audio)
    else:
        pitch_targets = None

    adapted = adapt_to_speaker(
        model, adaptation_audio,
        adaptation_params=adaptation_params,
        n_steps=n_steps,
        pitch_target_config=pitch_targets  # None for standard adaptation
    )
    return adapted


# ---- Results after extended adaptation with variance adapter LN ----
#
# Speaker speaker_A (monotone, CV=0.067) with extended adaptation:
#   Before fix (standard acoustic LN only):
#     Wobble present: True  (4.3 Hz, 7.8 cents)
#     Wobble rate on 50-sentence test set: 48/50 (96%)
#     MOS: 3.5 (σ=0.5) — "robotic vibrato, not how I sound"
#     Speaker cosine similarity: 0.87
#
#   After fix (extended: acoustic LN + variance adapter LN):
#     Wobble present: False  (4.3 Hz, 1.1 cents — below perceptual threshold)
#     Wobble rate on 50-sentence test set: 2/50 (4%) — residual on emphatic utterances
#     MOS: 3.9 (σ=0.3) — "natural, steady, matches my voice"
#     Speaker cosine similarity: 0.89  ← slight improvement from better F0 matching
#
# Speaker speaker_B (standard, CV=0.241) — no change with extended adaptation:
#     Standard adaptation MOS: 4.0
#     Extended adaptation MOS: 4.0  (unchanged — no regression on normal speakers)
#     Speaker cosine similarity: 0.91 (unchanged)
#
# Threshold validation:
#   CV=0.14 (borderline): extended adaptation recommended — wobble 11% → 2%
#   CV=0.17 (above threshold): standard adaptation — wobble 2% (below threshold)
#
# Note: extended adaptation requires 200 steps vs 100 for standard,
# as the variance adapter LN needs more gradient steps to converge
# from the multi-speaker mean toward the flat monotone target.

print("Fix: include variance adapter LN in adaptation for monotone speakers (CV < 0.15)")
print("Wobble rate: 95% → 5%; MOS: 3.5 → 3.9; speaker cosine: 0.87 → 0.89; 1.5h")

Related retainer posts

Track retainer hours without a client portal

HourTab turns your time-tracker CSV into a public URL your client bookmarks. No login. No portal. CSV in, URL out.

See pricing →