Vocos developer on retainer: 20–30 ms pre-ringing artifact preceding every voiced plosive when used as HiFi-GAN drop-in replacement — mel spectrogram frame rate mismatch between TTS pipeline and Vocos native training resolution causes ISTFT phase misalignment — resampling mel frames to Vocos native hop size eliminates pre-ringing — 3 weeks — Vocos developer on monthly retainer

Vocos (hubert-siuzdak/vocos) ISTFT-based neural vocoder • Python 3.11; PyTorch 2.1.x • real-time TTS pipeline • Vocos used as drop-in replacement for HiFi-GAN in FastSpeech2 TTS pipeline • FastSpeech2 outputs mel spectrograms at 16 kHz, hop size 200 (12.5 ms per frame) • Vocos generates clean audio for most phonemes but exhibits 20–30 ms pre-ringing artifact preceding every voiced plosive (/b/, /d/, /g/) • identical mel spectrograms fed to HiFi-GAN produce no pre-ringing • developer checks Vocos configuration, FastSpeech2 mel spectrogram parameters, and audio post-processing for 3 weeks • root cause: Vocos was trained with Complex STFT loss at 24 kHz, hop size 256 (≈10.67 ms per frame); FastSpeech2 TTS pipeline outputs mel spectrograms at 16 kHz, hop size 200 (12.5 ms per frame); mel frame duration mismatch (12.5 ms vs 10.67 ms) causes temporal interpolation during Vocos inference; Vocos ISTFT synthesis is phase-sensitive; phase misalignment accumulates across the closure period of voiced plosives where phase transition is sharp; Gibbs-phenomenon-like pre-ringing results • fix: resample TTS mel spectrogram to match Vocos native 24 kHz hop size 256 (10.67 ms frames) before Vocos inference • pre-ringing: 20–30 ms → 0 ms • MOS: 3.5 → 4.1 • 1.5h retainer engagement

Westfield Vocos TTS Ltd. built a real-time TTS pipeline for a voice assistant product using FastSpeech2 as the acoustic model. The pipeline originally used HiFi-GAN as the vocoder. When Vocos was released with its significantly lower latency and simpler ISTFT-based architecture, the team replaced HiFi-GAN with Vocos as a drop-in vocoder replacement to improve synthesis speed. For the majority of synthesized speech — vowels, fricatives, nasals, approximants — the Vocos output quality matched or exceeded HiFi-GAN. But a subset of synthesized outputs showed a consistent artifact: a 20–30 ms “pre-ringing” or “pre-echo” preceding every voiced plosive consonant (/b/, /d/, /g/). The artifact was an audible tone-like ringing that preceded the plosive burst by approximately 25 ms. It was clearly perceptible in MOS listening tests, and listeners described it as “sounds robotic” or “there’s an echo before the b”.

The team compared the same mel spectrograms fed to both HiFi-GAN and Vocos side-by-side. HiFi-GAN produced no pre-ringing on the same mels; the plosive bursts were clean. This confirmed the artifact was in the Vocos inference path, not in the FastSpeech2 acoustic model. They tried varying Vocos inference parameters including the number of mel channels, the FFT size, and the normalization mode — none had any effect on the pre-ringing. They also checked whether the FastSpeech2 mel spectrograms contained any pre-ringing in the spectrogram itself (they did not: the spectrogram showed clean near-silence in the closure period). After three weeks they engaged a Vocos retainer developer who identified the root cause as a mel frame duration mismatch between FastSpeech2’s 16 kHz/hop-200 configuration and Vocos’s native 24 kHz/hop-256 training resolution.

The TTS synthesis pipeline at the time of the engagement:

# Westfield Vocos TTS Ltd. — real-time TTS pipeline
# FastSpeech2 acoustic model → Vocos vocoder (drop-in HiFi-GAN replacement)
# Python 3.11, PyTorch 2.1.x

import torch
import numpy as np
import torchaudio
from fastspeech2 import FastSpeech2
from vocos import Vocos

# FastSpeech2 configuration (original, trained for 16 kHz output):
# sample_rate = 16000 Hz
# n_mels      = 80
# hop_size    = 200 samples  (200 / 16000 = 12.5 ms per mel frame)
# win_size    = 800 samples
# fmin        = 0 Hz
# fmax        = 8000 Hz (= Nyquist at 16 kHz)

# Vocos official checkpoint (hubert-siuzdak/vocos):
# Trained at: sample_rate = 24000 Hz
# n_mels      = 100
# hop_size    = 256 samples  (256 / 24000 ≈ 10.67 ms per mel frame)
# win_size    = 1024 samples
# fmin        = 0 Hz
# fmax        = 8000 Hz

fs2_model = FastSpeech2.from_pretrained("westfield-tts/fastspeech2-16k")
vocos     = Vocos.from_pretrained("hubert-siuzdak/vocos-mel-24khz")

def synthesize_buggy(text, speaker_id=0):
    """
    Synthesize speech with FastSpeech2 + Vocos (drop-in HiFi-GAN replacement).
    Bug: 20–30 ms pre-ringing before every voiced plosive (/b/, /d/, /g/).
    """
    with torch.inference_mode():
        # FastSpeech2: text → mel spectrogram
        # Output: mel [1, n_mels=80, T_frames] at 16 kHz / hop 200 (12.5 ms/frame)
        mel_16k = fs2_model(text, speaker_id=speaker_id)

        # Vocos: mel spectrogram → waveform via ISTFT
        # Problem: Vocos was trained on mels at 24 kHz / hop 256 (10.67 ms/frame)
        # FastSpeech2 outputs mels at 16 kHz / hop 200 (12.5 ms/frame)
        #
        # Frame duration mismatch: 12.5 ms vs 10.67 ms (17% longer frames from FastSpeech2)
        # Number of mel channels: 80 (FastSpeech2) vs 100 (Vocos) → n_mels mismatch too
        #
        # Vocos inference with mismatched mels:
        # 1. Vocos internally rescales the mel to its expected n_mels=100 (frequency axis)
        # 2. Vocos generates Fourier coefficients for each input frame
        #    at STFT resolution corresponding to its training (hop 256, win 1024 at 24 kHz)
        # 3. ISTFT reconstructs waveform at 24 kHz
        #
        # Phase misalignment at voiced plosive onsets:
        # - Voiced plosive closure: ~30 ms near-silence followed by sharp burst onset
        # - Sharp burst: rapid phase transition across all frequency bins simultaneously
        # - FastSpeech2 at 12.5 ms/frame: closure = ~2 frames of near-silence
        # - Vocos expects 10.67 ms/frame: same closure = ~2.8 frames
        # - Frame-boundary mismatch: Vocos receives 2 frames but expects 2.8
        # - 0.8 frames (~10 ms) of post-burst content is advanced into the closure frames
        # - ISTFT generates this pre-burst energy as pre-ringing before the actual burst
        # - Pre-ringing duration: 20–30 ms (consistent with 2× frame mismatch)

        waveform = vocos.decode(mel_16k)   # Bug: mismatched mel passed to Vocos
    return waveform

# Diagnosis: pre-ringing measurements across voiced plosive test set
# 50 utterances × 10 voiced plosive tokens per utterance = 500 plosive events
# Pre-ringing detector: energy in [t-50ms, t-5ms] window before burst onset
# Burst onset: detected via RMS energy spike > 20× local background (1 ms window)
#
# Pre-ringing measurements (HiFi-GAN baseline vs Vocos with mismatched mels):
#
#   Phoneme    HiFi-GAN pre-ringing (ms)   Vocos pre-ringing (ms)
#   /b/        0.8 ± 0.4                   24.3 ± 3.1   ← 24 ms pre-ringing
#   /d/        0.7 ± 0.3                   22.8 ± 2.9
#   /g/        0.9 ± 0.5                   27.1 ± 4.2   ← 27 ms pre-ringing
#   /p/ (ctrl) 0.8 ± 0.4                    1.2 ± 0.5   ← voiceless plosive: no pre-ringing
#   /t/ (ctrl) 0.7 ± 0.3                    1.0 ± 0.4   ← voiceless plosive: no pre-ringing
#
# Key observation: pre-ringing only on VOICED plosives (/b/,/d/,/g/), not voiceless (/p/,/t/,/k/).
# Explanation: voiced plosive closure is acoustically near-silence (voicing bar only),
# making the pre-ringing prominent. Voiceless plosive closure is complete silence —
# pre-ringing is present but below perceptual threshold.
#
# MOS evaluation (20-listener crowdsource, 50 utterances per condition):
#   HiFi-GAN baseline:     MOS 4.2 (σ=0.2)
#   Vocos (mismatched):    MOS 3.5 (σ=0.5) — "robotic echo", "ringing before b/d"

print("Root cause: mel frame duration mismatch 12.5 ms vs 10.67 ms")
print("ISTFT phase misalignment causes Gibbs-like pre-ringing at voiced plosive onsets")

Why mel frame rate mismatch causes ISTFT pre-ringing specifically at voiced plosives

Vocos replaces HiFi-GAN’s transposed convolution upsampling chain with an ISTFT-based synthesis approach. Instead of learning to upsample mel spectrogram features into a waveform through a series of strided convolutions, Vocos predicts complex Fourier coefficients (magnitude and phase) for each mel frame and reconstructs the waveform using the inverse short-time Fourier transform. This architecture is simpler, faster, and achieves comparable quality to HiFi-GAN when the mel spectrogram configuration matches the vocoder’s training configuration. The critical requirement is that the hop size, window size, and sample rate of the mel spectrogram must match the values used during Vocos training, because the Fourier coefficients Vocos generates are defined relative to the specific STFT frame size and overlap.

When FastSpeech2 outputs mel spectrograms at 16 kHz / hop 200 (12.5 ms per frame) and Vocos expects mels at 24 kHz / hop 256 (10.67 ms per frame), the temporal resolution mismatch (17% longer frames from FastSpeech2) means that each FastSpeech2 mel frame represents slightly more audio time than Vocos was trained to expect. When Vocos synthesizes a waveform, it generates Fourier coefficients for each input frame at the STFT resolution corresponding to 10.67 ms windows. The resulting waveform is therefore slightly stretched in time, and the phase alignment between adjacent STFT frames is computed assuming 10.67 ms frame spacing rather than the actual 12.5 ms spacing of the input mels. For smoothly-varying phonemes (vowels, fricatives, nasals), this phase misalignment is small enough that it manifests only as minor spectral coloring. But at voiced plosive onsets — where the acoustic waveform transitions from near-silence (the closure period) to a sudden burst with rapid phase change across all frequencies — the phase misalignment accumulates across the 2–3 closure frames and causes the ISTFT to generate energy in the temporal window preceding the burst. This is the pre-ringing artifact: the ISTFT is generating the onset phase transition slightly too early, spread across a 20–30 ms window before the actual burst.

The fix is to resample the FastSpeech2 mel spectrogram from its native 16 kHz / hop 200 configuration to match Vocos’s native 24 kHz / hop 256 configuration before Vocos inference. This requires resampling both the time axis (from 12.5 ms to 10.67 ms frames) and the frequency axis (from 80 mel channels to 100 mel channels), and ensuring the mel filter bank parameters (fmin, fmax) are consistent. After resampling, Vocos receives mel frames at the temporal resolution it was trained on, the phase alignment is correct, and the ISTFT generates the voiced plosive burst at the correct time without pre-ringing.

# Fix: resample TTS mel spectrogram to Vocos native resolution before inference
# Converts FastSpeech2 16 kHz / hop 200 / 80-mel to Vocos 24 kHz / hop 256 / 100-mel

import torch
import numpy as np
import torchaudio
import torchaudio.transforms as T
from fastspeech2 import FastSpeech2
from vocos import Vocos

fs2_model = FastSpeech2.from_pretrained("westfield-tts/fastspeech2-16k")
vocos     = Vocos.from_pretrained("hubert-siuzdak/vocos-mel-24khz")

# FastSpeech2 mel configuration
FS2_SR        = 16000
FS2_HOP       = 200    # 12.5 ms/frame
FS2_WIN       = 800
FS2_N_MELS    = 80
FS2_FMIN      = 0
FS2_FMAX      = 8000

# Vocos native mel configuration (must match Vocos training exactly)
VOCOS_SR      = 24000
VOCOS_HOP     = 256    # 10.67 ms/frame
VOCOS_WIN     = 1024
VOCOS_N_MELS  = 100
VOCOS_FMIN    = 0
VOCOS_FMAX    = 8000

def resample_mel_to_vocos(mel_fs2):
    """
    Resample FastSpeech2 mel spectrogram (16 kHz / hop 200 / 80-mel)
    to Vocos native resolution (24 kHz / hop 256 / 100-mel).

    mel_fs2: tensor [1, 80, T_fs2] — FastSpeech2 mel output
    Returns: tensor [1, 100, T_vocos] — Vocos-compatible mel

    Step 1: Frequency axis resampling (80 → 100 mel channels)
            Use linear interpolation along the mel frequency axis.
            Both configs use fmin=0, fmax=8000 — same mel filterbank range,
            just different number of filters. Linear interpolation is appropriate.

    Step 2: Time axis resampling (12.5 ms/frame → 10.67 ms/frame)
            Ratio: (10.67 ms / 12.5 ms) = 0.8533 → need 0.8533× as many frames
            Equivalently: (FS2_SR / FS2_HOP) / (VOCOS_SR / VOCOS_HOP)
                        = (16000/200) / (24000/256)
                        = 80 fps / 93.75 fps
                        → T_vocos = T_fs2 × (93.75 / 80) = T_fs2 × 1.172
            Use bilinear interpolation (preserve time-domain structure).
    """
    # Step 1: Frequency axis — 80 → 100 mel channels via linear interpolation
    # mel_fs2: [1, 80, T] → mel_100: [1, 100, T]
    mel_100 = torch.nn.functional.interpolate(
        mel_fs2.unsqueeze(0),          # [1, 1, 80, T]
        size=(VOCOS_N_MELS, mel_fs2.shape[-1]),
        mode='bilinear',
        align_corners=True,
    ).squeeze(0)                        # [1, 100, T]

    # Step 2: Time axis — 12.5 ms/frame → 10.67 ms/frame via bilinear interpolation
    fs2_fps   = FS2_SR / FS2_HOP       # 80.0 frames/second
    vocos_fps = VOCOS_SR / VOCOS_HOP   # 93.75 frames/second
    t_vocos   = int(round(mel_100.shape[-1] * vocos_fps / fs2_fps))

    mel_vocos = torch.nn.functional.interpolate(
        mel_100.unsqueeze(0),          # [1, 1, 100, T]
        size=(VOCOS_N_MELS, t_vocos),
        mode='bilinear',
        align_corners=True,
    ).squeeze(0)                        # [1, 100, T_vocos]

    return mel_vocos

def synthesize_fixed(text, speaker_id=0):
    """
    Synthesize speech with mel resampling to Vocos native resolution.
    Pre-ringing before voiced plosives eliminated.
    """
    with torch.inference_mode():
        # FastSpeech2: text → mel [1, 80, T] at 16 kHz / hop 200 / 12.5 ms/frame
        mel_fs2 = fs2_model(text, speaker_id=speaker_id)

        # Resample mel to Vocos native resolution (24 kHz / hop 256 / 100-mel)
        # This aligns the ISTFT phase relationships with Vocos training expectations
        mel_vocos = resample_mel_to_vocos(mel_fs2)

        # Vocos: Vocos-native mel → waveform via ISTFT (phase-consistent synthesis)
        waveform = vocos.decode(mel_vocos)   # Fixed: phase-aligned mel input
    return waveform

# ---- Results after mel resampling ----
#
# Pre-ringing measurements (500 voiced plosive events, same test set):
#
#   Phoneme    HiFi-GAN (ref)   Vocos (mismatch)   Vocos (fixed)
#   /b/        0.8 ± 0.4 ms     24.3 ± 3.1 ms      0.9 ± 0.4 ms  ← eliminated
#   /d/        0.7 ± 0.3 ms     22.8 ± 2.9 ms      0.8 ± 0.4 ms  ← eliminated
#   /g/        0.9 ± 0.5 ms     27.1 ± 4.2 ms      1.0 ± 0.5 ms  ← eliminated
#
# MOS evaluation (20-listener crowdsource, 50 utterances per condition):
#   HiFi-GAN baseline:          MOS 4.2 (σ=0.2)
#   Vocos (mismatched mels):    MOS 3.5 (σ=0.5)
#   Vocos (mel resampling fix): MOS 4.1 (σ=0.2) — matches HiFi-GAN
#
# Latency comparison (100 utterances, batch size 1, RTX 3090):
#   HiFi-GAN:         18.3 ms ± 1.2 ms per utterance
#   Vocos (fixed):     6.1 ms ± 0.4 ms per utterance  ← 3× faster than HiFi-GAN
#
# Mel resampling overhead:
#   bilinear interpolation (time + freq): < 0.5 ms per utterance
#   Net Vocos pipeline latency: 6.6 ms vs 18.3 ms (HiFi-GAN) → 2.8× speedup
#
# Note: for new TTS pipeline deployments, the preferred solution is to train
# FastSpeech2 (or the acoustic model) directly with Vocos-native mel parameters
# (24 kHz, hop 256, n_mels 100). The resampling fix is appropriate for retrofitting
# an existing trained acoustic model without retraining.

print("Fix: resample FastSpeech2 mel to Vocos native 24 kHz / hop 256 / 100-mel before inference")
print("Pre-ringing: 20–30 ms → 0 ms; MOS: 3.5 → 4.1; Vocos 2.8× faster than HiFi-GAN")

Related retainer posts

Track retainer hours without a client portal

HourTab turns your time-tracker CSV into a public URL your client bookmarks. No login. No portal. CSV in, URL out.

See pricing →