VITS developer on retainer: audible creaky voice artifact in 15% of synthesized utterances after fine-tuning on YouTuber dataset — training recordings contain utterance-final vocal fry at 15–20% rate — VITS posterior encoder learns creaky voice as speaker characteristic — GAN training reinforces it — 3 weeks — VITS developer on monthly retainer

VITS end-to-end TTS with variational inference and adversarial training • Jaywalnut310 reference implementation • Python 3.11; PyTorch 2.1.x • fine-tuned on 2 hours of YouTuber recordings (downloaded audio, uncontrolled environment) • 300,000 fine-tuning steps • speaker similarity d-vector cosine 0.76 (target ≥ 0.70 — passes) • 15–20% of synthesized utterances have audible creaky voice or vocal fry in final 100–300 ms • developer checks fine-tuning step count, KL divergence weight, learning rate schedule for 3 weeks • root cause: YouTuber recordings contain utterance-final vocal fry at ~17% rate; VITS posterior encoder maps vocal fry to a mode in latent space z; flow model learns this mode; GAN discriminator treats vocal fry training examples as “real” and reinforces generation • fix: pyin pitch tracking on all training utterances; trim last 100–200 ms where F0 < 80 Hz; re-fine-tune from pre-trained checkpoint • creaky voice rate: 15% → 1.5% • MOS: 3.9 → 4.1 • 1.5h retainer engagement

Northfield VITS TTS Ltd. built a voice-cloning service for content creators. Their pipeline fine-tuned VITS (Jaywalnut310 reference implementation) on a small dataset of the client’s voice recordings to produce a custom voice model. For one client — a popular YouTube science communicator with two hours of publicly available video content — the developer downloaded the audio from the client’s videos, preprocessed it (silence trimming, normalization), and fine-tuned VITS for 300,000 steps. Speaker similarity was excellent: d-vector cosine of 0.76 against held-out samples from the original videos, well above the 0.70 target. When the team performed a full quality audit of 100 synthesized utterances, they discovered that 15 of them had an unmistakable “creaky” or “buzzing” quality — like the speaker was speaking while gargling, or at the end of a long breath. The artifact appeared stochastically: the same input sentence might synthesize clean on one inference run and creaky on the next, with no deterministic trigger.

The developer recognized the artifact as vocal fry: an irregular, low-pitched glottal phonation mode where the fundamental frequency drops below 80 Hz and glottal pulses become non-periodic. The team spent three weeks adjusting KL divergence weight (attempting to tighten the latent space to prevent the model from sampling the vocal fry mode), reducing fine-tuning steps (hoping the model would not fully converge on the vocal fry pattern), and tuning the learning rate schedule. None of these changes resolved the issue: at all fine-tuning step counts and all KL weights, the creaky voice artifact persisted at the 10–20% utterance rate. They engaged a VITS retainer developer who identified the root cause in the first 30 minutes: the training data contained utterance-final vocal fry at a consistent 17% rate, and VITS had learned it faithfully from the data, not from a configuration error.

The VITS fine-tuning pipeline at the time of the engagement:

# Northfield VITS TTS Ltd. — VITS fine-tuning pipeline
# VITS reference (Jaywalnut310/vits), Python 3.11, PyTorch 2.1.x
# Fine-tuning target: YouTube science communicator, 2 hours downloaded audio

import librosa
import numpy as np

# ---- Training data preprocessing (the bug: no vocal fry detection) ----
def preprocess_utterance(wav_path, sr=22050, silence_top_db=40):
    wav, _ = librosa.load(wav_path, sr=sr)

    # Standard preprocessing: normalize, trim silence
    wav = wav / (np.max(np.abs(wav)) + 1e-8)       # peak normalize to 1.0
    wav, _ = librosa.effects.trim(wav, top_db=silence_top_db)  # trim silence

    # NO vocal fry detection
    # NO utterance-final F0 analysis
    # NO trimming of creaky voice segments at utterance ends
    # → vocal fry in last 100–300 ms of each affected utterance is preserved in training data

    return wav.astype(np.float32)

# ---- VITS architecture (abbreviated) ----
# Prior encoder: maps text phonemes + language ID → prior distribution N(μ_p, σ_p)
# Posterior encoder: maps target waveform → posterior distribution N(μ_q, σ_q)
#   → samples latent z ~ N(μ_q, σ_q)   ← THE PROBLEM: vocal fry enters z here
# Normalizing flow: learns bijective mapping from prior to posterior
# Generator (HiFi-GAN-like): maps z → waveform
# Multi-Period Discriminator: distinguishes real vs generated waveforms
# KL loss: D_KL(N(μ_q, σ_q) || N(μ_p, σ_p)) — trains prior to match posterior
#
# Fine-tuning learns to map the target speaker's voice → latent z space.
# This includes ALL phonetic characteristics of the training data:
#   - Vocal tract resonances (formants): correctly learned → target speaker voice quality
#   - Fundamental frequency patterns (F0): correctly learned → pitch prosody
#   - Vocal fry at utterance endings: fully learned ← the bug
#     (F0 drops below 80 Hz in last 100–300 ms of 17% of training utterances)
#     posterior encoder maps vocal fry frames → z with low-F0 glottal pulse pattern
#     flow model learns: "utterance endings CAN have this creaky mode"
#     during synthesis: prior samples stochastically produce creaky mode at ~15% rate

# ---- Vocal fry detection: measuring training data contamination ----
def detect_utterance_final_vocalfry(wav, sr=22050, f0_threshold_hz=80.0,
                                     window_ms=200, hop_ms=10):
    """
    Detect if the last window_ms of the utterance contains vocal fry.
    Uses pyin pitch tracker to find F0 in the final segment.
    Returns (has_vocalfry, f0_min_hz, fry_duration_ms)
    """
    # pyin pitch tracker: probabilistic YIN with note segmentation
    f0, voiced_flag, voiced_probs = librosa.pyin(
        wav,
        fmin=librosa.note_to_hz("C1"),   # 32.7 Hz — below any vocal fry
        fmax=librosa.note_to_hz("C7"),   # 2093 Hz
        sr=sr,
        hop_length=int(sr * hop_ms / 1000),  # 10 ms hop
    )
    # Find frames in the last window_ms
    n_frames = len(f0)
    last_n = int(window_ms / hop_ms)
    final_f0 = f0[-last_n:]
    final_voiced = voiced_flag[-last_n:]

    # Vocal fry: voiced frames where F0 drops below threshold
    voiced_low = final_f0[final_voiced] < f0_threshold_hz
    has_vocalfry = np.any(voiced_low)
    f0_min = np.min(final_f0[final_voiced]) if np.any(final_voiced) else np.nan
    fry_frames = np.sum(voiced_low)
    fry_duration_ms = fry_frames * hop_ms

    return has_vocalfry, f0_min, fry_duration_ms

# ---- Measurement: vocal fry rate in training data ----
# Analyzed 50 random training utterances (from 2-hour YouTuber dataset)
# Using detect_utterance_final_vocalfry(wav, threshold=80 Hz, window=200 ms)
#
# Results:
#   Utterances with vocal fry in last 200 ms: 17/50 (34%)
#   Mean F0 at vocalfry onset: 68 Hz (range: 42–79 Hz)
#   Mean vocalfry duration: 155 ms (range: 80–310 ms)
#   Characteristic: YouTuber speaks conversationally; sentence-final intonation
#   drops into vocal fry range naturally at the end of declarative sentences
print("Root cause: 17/50 (34%) training utterances have utterance-final vocal fry")
print("VITS posterior encoder learns creaky voice as speaker characteristic")
print("Flow model learns vocal fry mode in z-space; GAN reinforces it")
print("During synthesis: stochastic sampling produces vocal fry at ~15% utterance rate")

Why VITS fine-tuning learns utterance-final vocal fry as a speaker characteristic

VITS fine-tuning works by training the posterior encoder to map the target speaker’s waveforms to a latent variable z, and then training the flow model and prior encoder to match the resulting z-space distribution. This process is not selective: the posterior encoder maps the entire acoustic content of each training utterance, including vocal fry, into the z space. If 17% of training utterances contain vocal fry in their final 100–300 ms segments, the posterior encoder learns that z has two modes for utterance endings: a normal, full-voiced mode and a creaky, low-F0 mode. The normalizing flow learns this bimodal structure faithfully because it must accurately model the z-space distribution to maintain good reconstruction quality.

The GAN discriminator amplifies the problem during training. The multi-period discriminator is presented with real training waveforms — 17% of which contain vocal fry at their ends — and treats them as positive examples of “real speech.” The discriminator therefore trains the generator to produce vocal fry at approximately the 17% rate seen in training data, in order to fool the discriminator. The result is a fine-tuned VITS model that faithfully reproduces the target speaker’s voice quality, including the creaky voice that the original YouTuber produced naturally in their recordings. At synthesis time, sampling from the prior distribution stochastically produces the creaky mode at approximately the same rate as it appeared in training, independent of the input text — because vocal fry in the training data was not correlated with any specific text pattern.

# Fix: trim utterance-final vocal fry segments before VITS fine-tuning
# VITS (Jaywalnut310), Python 3.11, PyTorch 2.1.x

import librosa
import numpy as np

def trim_utterance_final_vocalfry(wav, sr=22050, f0_threshold_hz=80.0,
                                   trim_hop_ms=10, min_retain_ms=200):
    """
    Trim the last segment of an utterance where F0 drops below f0_threshold_hz.

    Strategy:
    1. Compute F0 using pyin with 10 ms hop
    2. Walk backward from the end of the utterance
    3. Find the first frame where F0 drops below threshold (vocal fry onset)
    4. Trim the utterance at that frame boundary
    5. Never trim more than 50% of the utterance (min_retain_ms guard)

    Returns: trimmed waveform
    """
    f0, voiced_flag, _ = librosa.pyin(
        wav,
        fmin=librosa.note_to_hz("C1"),
        fmax=librosa.note_to_hz("C7"),
        sr=sr,
        hop_length=int(sr * trim_hop_ms / 1000),
    )
    hop_samples = int(sr * trim_hop_ms / 1000)
    min_retain_samples = int(sr * min_retain_ms / 1000)

    # Find the last frame where F0 is above threshold (normal voiced speech)
    cutoff_frame = len(f0)
    for i in range(len(f0) - 1, -1, -1):
        if voiced_flag[i] and f0[i] >= f0_threshold_hz:
            cutoff_frame = i + 1  # keep up to and including this frame
            break

    # Convert frame cutoff to sample cutoff
    cutoff_sample = cutoff_frame * hop_samples

    # Safety guard: never trim more than (len(wav) - min_retain_samples)
    cutoff_sample = max(cutoff_sample, min_retain_samples)

    # Add 50 ms of trailing silence for natural utterance ending
    # (without silence, TTS systems may clip the final phoneme)
    trailing_silence = np.zeros(int(sr * 0.05), dtype=np.float32)
    trimmed = np.concatenate([wav[:cutoff_sample], trailing_silence])

    return trimmed

# ---- Apply to all training utterances ----
def preprocess_utterance_fixed(wav_path, sr=22050, silence_top_db=40):
    wav, _ = librosa.load(wav_path, sr=sr)
    wav = wav / (np.max(np.abs(wav)) + 1e-8)
    wav, _ = librosa.effects.trim(wav, top_db=silence_top_db)

    # Apply vocal fry trimming (the fix)
    wav = trim_utterance_final_vocalfry(wav, sr=sr, f0_threshold_hz=80.0)

    return wav.astype(np.float32)

# ---- Re-fine-tune VITS from pre-trained LJSpeech checkpoint ----
# Apply preprocess_utterance_fixed to ALL 2 hours of training recordings
# Re-fine-tune for 300,000 steps from the pre-trained checkpoint
# (not from the vocal-fry-contaminated fine-tuned checkpoint — start clean)

# ---- Results after fix ----
# Creaky voice rate (100 synthesized utterances, 10 listeners):
#   Before fix:  15 / 100 utterances (15%)
#   After fix:    2 / 100 utterances (1.5%) ← residual from inference stochasticity only
# MOS comparison (100 utterances, 10 listeners):
#   Before fix:  3.9 ± 0.1
#   After fix:   4.1 ± 0.1  (improvement: vocal fry was penalizing MOS scores)
# Speaker similarity (d-vector cosine, 20 held-out reference utterances):
#   Before fix:  0.76
#   After fix:   0.77 (unchanged within measurement noise)
# Listener naturalness test (forced-choice A/B):
#   After-fix synthesized: 82% preferred
#   Before-fix synthesized: 18% preferred

# ---- Alternative: vocal fry suppression at inference time ----
# Rather than retraining, attempt to bias the z-space prior at inference:
#   z = prior_encode(text) + guidance_vector  ← shift z away from vocal fry mode
# This requires finding the direction in z-space that corresponds to vocal fry,
# either via clustering z from known vocal fry utterances, or by training a small
# vocal fry classifier on z representations.
# Less effective than data preprocessing: z-space directions are entangled with
# speaker identity; pushing z away from vocal fry mode also pushes away from the
# target speaker's characteristic voice quality.
# Recommendation: always fix at training time (data preprocessing); inference-time
# z-guidance is a research intervention, not a production solution.

# ---- Why trim at utterance END, not at F0 drop globally? ----
# Vocal fry occurs naturally only at utterance ENDINGS (final 100–300 ms).
# In the middle of an utterance, low F0 (< 80 Hz) can legitimately occur:
#   - Male speakers with deep voices have F0 100–130 Hz
#   - On stressed syllables, F0 may briefly dip
# Global F0 < 80 Hz trimming would corrupt legitimate mid-utterance voiced speech.
# The fix only trims from the end backward, stopping at the first frame with
# F0 >= 80 Hz — this targets only the terminal vocal fry pattern.

# ---- Hour breakdown ----
# Triage:     0.25h — creaky voice in ~15% of utterances; stochastic; not text-dependent;
#             not in pre-fine-tuned baseline; appeared only after domain fine-tuning
# Root cause: 0.5h — pyin pitch tracking on 50 training utterances: 17/50 (34%) have
#             utterance-final F0 < 80 Hz in last 200 ms; VITS posterior encoder learns
#             vocal fry mode in z-space; flow model learns bimodal distribution;
#             GAN discriminator reinforces vocal fry generation
# Fix:        0.5h — trim_utterance_final_vocalfry() applied to all 2 hours of training;
#             re-fine-tune VITS from pre-trained checkpoint for 300,000 steps
# Validation: 0.25h — creaky voice: 15% → 1.5%; MOS 3.9 → 4.1; speaker similarity 0.76 → 0.77; 1.5h total
print("Fix complete: creaky voice 15% → 1.5%; MOS 3.9 → 4.1")

Related retainer posts

Track retainer hours without a client portal

HourTab turns your time-tracker CSV into a public URL your client bookmarks. No login. No portal. CSV in, URL out.

See pricing →