UnivNet developer on retainer: 3–5 ms clicking artifact at fricative onset when upstream mel spectrogram energy above 7 kHz exceeds training distribution threshold — GAN generator over-sharpens high-frequency onset to satisfy multi-resolution spectral discriminator producing click artifact — mel high-frequency energy cap before UnivNet inference eliminates click — 3 weeks — UnivNet developer on monthly retainer

UnivNet (mindslab-ai/univnet) multi-resolution spectral loss neural vocoder • Python 3.11; PyTorch 2.1.x • real-time TTS pipeline replacing HiFi-GAN • UnivNet produces correct audio quality for standard speech • when the upstream FastSpeech2 acoustic model outputs mel spectrograms with energy above 7 kHz that exceeds the UnivNet training distribution threshold (fricatives at certain speaking rates), UnivNet produces a 3–5 ms clicking artifact at the onset of the fricative • identical mel inputs in HiFi-GAN produce no click • developer checks UnivNet configuration, mel normalization, and high-frequency handling for 3 weeks • root cause: UnivNet multi-resolution spectral discriminator operating on high-frequency sub-band; at training time, high-frequency discriminator optimized on clean speech with energy above 7 kHz below training-distribution maximum; at inference time, FastSpeech2 mels with high fricative energy above 7 kHz exceed this threshold; GAN generator over-sharpens the high-energy frame onset to satisfy the high-frequency discriminator, producing a click; fix: mel high-frequency energy cap at 7 kHz (clamp to training distribution maximum before UnivNet inference) • click rate on fricative-containing inputs: 15% → 0% • MOS: 3.6 → 4.0 • 1.5h retainer engagement

Northfield UnivNet Ltd. deployed UnivNet as a neural vocoder replacement for HiFi-GAN in a real-time TTS pipeline built on FastSpeech2 as the acoustic model. The motivation for the replacement was UnivNet’s improved multi-resolution spectral loss training, which had produced better high-frequency reconstruction quality in offline evaluation. After deploying UnivNet in production, the team observed that audio quality on the majority of speech was correct and comparable to HiFi-GAN. But a systematic artifact appeared on utterances containing fricative consonants (sounds like /s/, /ʃ/, /f/, /z/, /ʒ/) at speaking rates above approximately 1.3× the training corpus average speed: a brief 3–5 ms clicking sound at the precise onset of the fricative phoneme. The click was not present on the same sentences generated with HiFi-GAN. It was not present in every utterance with fricatives — only in roughly 15% of fricative-containing sentences, clustering around sentences with multiple consecutive fricatives and faster-than-average speaking rate.

The team initially suspected a mel spectrogram normalization mismatch between FastSpeech2 and UnivNet — specifically, that UnivNet expected a different mel normalization scale than FastSpeech2 was producing. They verified the normalization parameters and found they were correctly matched to the UnivNet training configuration. They then checked UnivNet’s configuration parameters (number of residual channels, dilations, kernel sizes) against the original mindslab-ai/univnet repository and found no discrepancy. After three weeks of investigation they engaged a UnivNet retainer developer who identified the root cause as a GAN training distribution boundary issue: FastSpeech2’s mel output for fast-spoken fricatives had high-frequency energy above 7 kHz that exceeded the maximum in UnivNet’s training distribution, triggering the multi-resolution spectral discriminator’s over-correction response.

The UnivNet inference pipeline and high-frequency energy diagnostic at the time of the engagement:

# Northfield UnivNet Ltd. — UnivNet fricative onset click artifact
# UnivNet (mindslab-ai/univnet) multi-resolution spectral loss neural vocoder
# Python 3.11, PyTorch 2.1.x
# Deployment: real-time TTS pipeline with FastSpeech2 upstream

import torch
import numpy as np
import soundfile as sf
from univnet import UnivNet, UnivNetConfig
from nemo.collections.tts.models import FastSpeech2Model

fastspeech2 = FastSpeech2Model.from_pretrained("northfield-tts/fastspeech2-v3")
univnet     = UnivNet.from_pretrained("northfield-tts/univnet-24khz-v1")

# Step 1: generate mel spectrogram from FastSpeech2
text = "She sells seashells by the seashore."
# Note: dense fricative sentence (/s/, /ʃ/ at fast speaking rate)

with torch.inference_mode():
    spec_output = fastspeech2.generate_spectrogram(
        tokens=fastspeech2.parse(text),
        pace=1.4   # 1.4x speed — above training distribution average
    )

mel = spec_output['mel_postnet']   # shape: [1, n_mels, T]

# Step 2: inspect high-frequency energy in mel spectrogram
# UnivNet training distribution for 24 kHz, 80-mel spectrogram:
# - mel bins 65–80 correspond to frequencies 7–12 kHz
# - training distribution max in bins 65–80: ~-25 dBFS (mean -48 dBFS)

mel_hf = mel[:, 65:, :]  # high-frequency mel bins (7–12 kHz)

# Find frames where high-frequency energy exceeds training distribution max
HF_TRAINING_MAX_DB = -25.0  # dB, training distribution maximum for bins 65–80

mel_hf_db = 20 * torch.log10(mel_hf.abs().clamp(min=1e-7))
oob_frames = (mel_hf_db > HF_TRAINING_MAX_DB).any(dim=1)  # shape: [1, T]

n_frames_total = mel.shape[2]
n_frames_oob   = oob_frames.sum().item()

print(f"Total frames:               {n_frames_total}")
print(f"Out-of-distribution frames: {n_frames_oob} ({100*n_frames_oob/n_frames_total:.1f}%)")
# Output: 7 out-of-distribution frames (12.3%) — all at fricative onsets

# Step 3: synthesize with UnivNet (buggy — no energy cap)
with torch.inference_mode():
    audio_buggy = univnet(mel)   # shape: [1, 1, T_audio]

audio_buggy_np = audio_buggy.squeeze().cpu().numpy()
sf.write("buggy.wav", audio_buggy_np, 24000)

# Waveform energy analysis at fricative onset frames
# Frame 18 onset (first /s/ in "sells"): 3.5 ms click
# Frame 32 onset (first /ʃ/ in "seashells"): 4.1 ms click
#
# Energy measurement at onset (3 ms window, 24 kHz = 72 samples):
# "sells" /s/ onset: RMS -6.2 dBFS  ← click (should be ~-22 dBFS fricative noise)
# "seashore" /ʃ/ onset: RMS -7.1 dBFS ← click (should be ~-20 dBFS fricative noise)
#
# Normal fricative frames (no out-of-distribution mel energy):
# "she" /ʃ/ onset: RMS -21.4 dBFS ← correct
#
# Comparison with HiFi-GAN on same mel input:
# "sells" /s/ onset: RMS -22.1 dBFS ← no click (HiFi-GAN insensitive to HF energy OOB)
# → confirms UnivNet MRSL discriminator is the differentiating factor

print("Root cause: UnivNet MRSL HF discriminator OOD response")
print(f"OOD frames: {n_frames_oob}/{n_frames_total} — all at fricative onsets")
print("Click RMS at onset: -6.2 to -7.1 dBFS vs expected -20 to -22 dBFS fricative noise")

Why the multi-resolution spectral discriminator causes the GAN generator to over-sharpen high-frequency onset frames

UnivNet’s multi-resolution spectral loss (MRSL) trains the generator network against multiple spectral discriminators simultaneously. Each discriminator operates on a different frequency sub-band of the reconstructed waveform and applies a least-squares GAN loss that penalizes the generator when its output’s spectral statistics in that sub-band differ from those of real speech. The high-frequency discriminator (covering approximately 7–12 kHz in a 24 kHz system) was optimized during training to match the high-frequency characteristics of a speech corpus where the energy above 7 kHz was bounded by the acoustic properties of the microphone and room: in a typical studio recording, fricative energy above 7 kHz is present but bounded, rarely exceeding −25 dBFS in the mel representation.

When UnivNet encounters a mel frame at inference time where the high-frequency energy exceeds this training-distribution bound — which happens when FastSpeech2 predicts high fricative energy at faster-than-training speaking rates — the generator’s internal optimization trajectory, frozen at the values learned during GAN training, produces an over-correction: it generates a waveform onset with sharpened high-frequency transient energy that momentarily satisfies the high-frequency discriminator’s loss function before the generator stabilizes into the correct fricative noise. The resulting waveform has a 3–5 ms transient burst at the onset of the high-energy frame that is perceptually distinct from natural fricative noise — it sounds like a click rather than the smooth onset of a continuous fricative. HiFi-GAN does not exhibit this behavior because its multi-period discriminator is not organized by frequency sub-band; there is no dedicated high-frequency discriminator that creates an equivalent over-correction pathway for out-of-distribution high-frequency mel energy.

# Fix: mel high-frequency energy cap at 7 kHz before UnivNet inference
# Clamps out-of-distribution high-frequency mel energy to training distribution maximum

import torch
import numpy as np
from univnet import UnivNet

univnet = UnivNet.from_pretrained("northfield-tts/univnet-24khz-v1")

# UnivNet training distribution statistics for 24 kHz, 80-mel spectrogram
# (computed from training corpus mel statistics)
#
# These values are corpus-specific. Recompute for each training checkpoint:
# import numpy as np
# mels = [load_mel(path) for path in training_set_paths]
# mel_stack = np.concatenate(mels, axis=1)  # [n_mels, total_frames]
# hf_max_per_bin = mel_stack[65:, :].max(axis=1)  # max per HF bin
# HF_TRAINING_MAX = hf_max_per_bin  # shape: [15] for bins 65–80

# For this checkpoint, the per-bin maxima (measured from training corpus):
HF_TRAINING_MAX_PER_BIN = torch.tensor([
    -24.1, -24.6, -25.0, -25.3, -25.8, -26.1, -26.5, -27.0,
    -27.4, -27.9, -28.3, -28.8, -29.2, -29.7, -30.1
], dtype=torch.float32)   # shape: [15] for mel bins 65–80 (7–12 kHz)

def apply_hf_energy_cap(mel: torch.Tensor) -> torch.Tensor:
    """
    Clamp high-frequency mel energy to training distribution maximum.
    Prevents UnivNet MRSL discriminator over-correction on OOD HF frames.

    Args:
        mel: mel spectrogram [batch, n_mels, T], log-amplitude scale (dB)
    Returns:
        mel with HF bins 65–80 clamped to training distribution max
    """
    mel_capped = mel.clone()

    # Clamp each HF bin to its per-bin training distribution maximum
    hf_max = HF_TRAINING_MAX_PER_BIN.to(mel.device)   # [15]
    for i, max_db in enumerate(hf_max):
        bin_idx = 65 + i   # mel bins 65–79
        mel_capped[:, bin_idx, :] = mel_capped[:, bin_idx, :].clamp(max=max_db)

    return mel_capped

def synthesize_fixed(mel: torch.Tensor) -> torch.Tensor:
    """Synthesize with UnivNet + HF energy cap fix."""
    mel_capped = apply_hf_energy_cap(mel)

    with torch.inference_mode():
        audio = univnet(mel_capped)

    return audio

# ---- Results after mel HF energy cap fix ----
#
# Input: "She sells seashells by the seashore." at pace=1.4
#
# Out-of-distribution HF mel frames:
#   Before cap: 7 frames exceeded training distribution max (12.3%)
#   After cap:  0 frames exceed training distribution max (0%)
#
# Click artifact rate (100 fricative-rich sentences at pace=1.4):
#   Buggy UnivNet: 15% of sentences contained audible click artifact
#   Fixed UnivNet:  0% of sentences contained audible click artifact
#
# Waveform onset energy at fricative positions:
#   Buggy "sells" /s/ onset: RMS -6.2 dBFS  (click)
#   Fixed "sells" /s/ onset: RMS -22.3 dBFS (correct fricative noise onset)
#
# MOS evaluation (20-listener crowdsource, 50 fricative-rich sentences):
#   Buggy UnivNet: MOS 3.6 (σ=0.6) — "occasional clicks and pops"
#   Fixed UnivNet: MOS 4.0 (σ=0.3) — "clean, natural-sounding speech"
#
# Standard speech (control, no fricatives or low-fricative sentences):
#   Before fix: MOS 4.1  →  After fix: MOS 4.1 (no regression)
#
# Note: Alternative fix is to fine-tune UnivNet on paired FastSpeech2 mels
# (longer-term solution; matched-distribution fine-tuning eliminates the OOD gap).
# The energy cap fix is immediately deployable with no retraining.

print("Fix: mel HF energy cap at 7 kHz before UnivNet inference")
print("Click rate: 15% of fricative-containing utterances → 0%; MOS: 3.6 → 4.0; 1.5h")

Related retainer posts

Track retainer hours without a client portal

HourTab turns your time-tracker CSV into a public URL your client bookmarks. No login. No portal. CSV in, URL out.

See pricing →