Matcha-TTS developer on retainer: click artifact every ~3 seconds after fine-tuning on Zoom call recordings — Zoom DSP 33 ms frame boundary discontinuities learned by flow model — individually inaudible artifacts accumulate to audible click — 3 weeks — Matcha-TTS developer on monthly retainer

Matcha-TTS (shivammehta007/Matcha-TTS) flow-matching TTS • Python 3.11; PyTorch 2.1.x • corporate narration service • fine-tuned on 30 min Zoom call recordings of corporate client voice • base model: clean, no artifacts • fine-tuned model: correct speaker identity but intermittent 10–20 ms click/discontinuity at ~3-second intervals • developer checks fine-tuning steps, ODE solver steps, temperature for 3 weeks • root cause: Zoom DSP noise suppression operates on 33 ms frames; frame-boundary amplitude discontinuities (<−50 dBFS) in every training recording; flow model learns 33 ms periodic spectral pattern as speaker characteristic; synthesis reproduces it; artifacts accumulate to audible click every ~3 s • fix: DeepFilterNet neural enhancement on all Zoom recordings before mel extraction; re-fine-tune • click artifacts eliminated • MOS 3.8 → 4.0 • 1.5h retainer engagement

Eastfield Matcha TTS Ltd. built a corporate narration service that adapts Matcha-TTS to each client’s voice, allowing companies to generate internal training videos and product documentation in their chosen spokesperson’s voice. The service collected client voice recordings from Zoom meeting recordings — practical for clients who didn’t have access to a recording studio. The base Matcha-TTS model, before fine-tuning, produced excellent quality synthesis. After fine-tuning on a client’s Zoom recordings, the model produced the client’s voice accurately — but all synthesized output had a persistent artifact: a brief click or waveform discontinuity of approximately 10–20 ms appearing at regular intervals of about 3 seconds. The artifact was subtle enough to be missed on a quick listen but became distracting in narration segments longer than 30 seconds.

The team tried reducing the fine-tuning step count, thinking the model had overfit to some artifact in the training data. Fewer steps reduced speaker similarity without changing the click interval. They increased the ODE solver steps from the default 10 to 25 and 50, which improved overall synthesis quality slightly but did not affect the click rate. They tried changing the ODE solver type (Euler to RK4). They examined the training data by playing it back — it sounded clean, with no audible clicks. They checked for silent frames, clipping, and DC offset. After three weeks of investigation, they engaged a Matcha-TTS retainer developer who identified the Zoom DSP frame boundary artifact as the source, invisible to casual listening but measurable at the mel spectrogram level and reproducible by the flow model.

The Matcha-TTS fine-tuning pipeline at the time of the engagement:

# Eastfield Matcha TTS Ltd. — Matcha-TTS fine-tuning pipeline
# Matcha-TTS (shivammehta007/Matcha-TTS), Python 3.11, PyTorch 2.1.x
# Use case: voice adaptation to corporate spokesperson from Zoom recordings

import torch
import librosa
import numpy as np

# ---- Training data: Zoom recordings (the bug) ----
#
# Training data: 30 minutes of Zoom meeting recordings, 44.1 kHz stereo → 22.05 kHz mono
# Zoom noise suppression and echo cancellation: ENABLED (default Zoom settings)
#
# Zoom DSP architecture (reverse-engineered from waveform analysis):
#   Zoom processes audio in fixed 33 ms frames (10 ms lookahead + 23 ms frame)
#   At each 33 ms boundary, Zoom's DSP:
#     1. Updates noise floor estimate from previous frame
#     2. Applies spectral subtraction with updated estimate
#     3. Outputs processed frame
#   Frame boundary transition: amplitude discontinuity < -50 dBFS
#   Individually inaudible (< -50 dBFS), but present in every 33 ms frame boundary
#
# Mel spectrogram of Zoom training recording (22.05 kHz, hop_size=256 → 11.6 ms frame):
#   Every 3 mel frames (3 × 11.6 ms ≈ 34.8 ms ≈ 1 Zoom DSP frame):
#     Mel bin 0 (0-275 Hz):  amplitude dip of 0.8-1.2 dB
#     Mel bin 1 (275-550 Hz): amplitude dip of 0.4-0.7 dB
#   Pattern repeats at 33 ms intervals throughout all 30 minutes of training data
#   Consistent enough that flow model optimizer treats it as a speaker characteristic

def extract_mel_buggy(wav_path, sr=22050, n_fft=1024, hop_length=256, n_mels=80):
    """Extract mel spectrogram from Zoom recording (no artifact removal)."""
    audio, file_sr = librosa.load(wav_path, sr=sr)
    # No preprocessing — Zoom DSP artifacts present in raw audio
    mel = librosa.feature.melspectrogram(y=audio, sr=sr, n_fft=n_fft,
                                          hop_length=hop_length, n_mels=n_mels)
    return librosa.power_to_db(mel, ref=np.max)

# ---- Artifact analysis (how root cause was confirmed) ----
# Apply autocorrelation to mel spectrogram to find periodic structure
#
# import numpy as np
# mel_spectrogram = extract_mel_buggy("client_zoom_recording.wav")  # shape: (80, T)
#
# Analyze periodicity in mel bin 0 (most affected by Zoom DSP) over time:
#   mel_bin_0_energy = mel_spectrogram[0, :]  # (T,)
#   autocorr = np.correlate(mel_bin_0_energy, mel_bin_0_energy, mode='full')
#   autocorr = autocorr[len(autocorr)//2:]
#
# Autocorrelation peaks:
#   Lag 0:    1.000 (self)
#   Lag 3:    0.312 (3 mel frames = 34.8 ms ≈ 1 Zoom DSP frame)
#   Lag 6:    0.287 (2 Zoom frames)
#   Lag 9:    0.271 (3 Zoom frames)
#   ...
#   Lag 258:  0.189 (1 second = 86 mel frames, harmonic of 33 ms period)
#
# Zoom 33 ms frame boundary artifact confirmed in mel spectrogram
# Flow model fine-tuning optimizes ODE trajectory to reproduce this periodic spectral pattern
# During synthesis, flow model generates mel frames with the same 33 ms periodic dip
# 100 × 33 ms = 3,300 ms ≈ 3 seconds → audible click accumulates at ~3-second intervals
print("Root cause: Zoom DSP 33 ms frame boundaries in mel spectrogram")
print("Flow model learns periodic spectral dip as speaker characteristic")
print("Synthesis reproduces 33 ms artifact; audible click accumulates every ~3 seconds")

Why Matcha-TTS flow model learns Zoom DSP frame artifacts as speaker characteristics

Matcha-TTS uses a continuous normalizing flow (CNF) trained with optimal transport flow matching to learn a deterministic mapping from a Gaussian noise prior to the mel spectrogram of the target speaker. During fine-tuning, the flow model updates its parameters to minimize the distance between generated mel spectrograms and the target mel spectrograms extracted from the training recordings. If the training mel spectrograms contain a periodic spectral pattern — even one that is individually very subtle (a 0.8 dB amplitude dip in the lowest mel bins at every 33 ms frame boundary) — and that pattern is consistent enough across all training examples, the flow model’s optimizer will incorporate it into the learned trajectory. The model cannot distinguish between spectral patterns that originate from the speaker’s vocal tract (the target voice characteristic) and patterns that originate from the recording chain (Zoom DSP artifact). Both are present in every training mel frame, and both reduce the flow matching objective when reproduced in synthesis.

The fix requires removing the Zoom DSP artifact from the training recordings before mel spectrogram extraction, so the flow model is trained only on the speaker’s actual vocal characteristics. DeepFilterNet (a neural audio enhancement model specifically trained to remove noise and DSP processing artifacts from real-world recordings) processes the Zoom recordings and outputs a clean waveform with the frame-boundary discontinuities removed. After enhancement, the mel spectrogram periodicity at 33 ms is eliminated, and re-fine-tuning from the pre-trained checkpoint produces a speaker-adapted model that generates clean synthesis without the click artifact. An alternative is Demucs audio enhancement, which also removes DSP artifacts from Zoom recordings effectively. The key is that any neural enhancement model must be applied before mel extraction, not as a post-processing step on the synthesized output — post-processing cannot correct artifacts that are already embedded in the flow model’s parameters.

# Fix: DeepFilterNet audio enhancement before mel extraction and fine-tuning
# Matcha-TTS (shivammehta007/Matcha-TTS), Python 3.11, PyTorch 2.1.x

import torch
import numpy as np
import soundfile as sf
import librosa
from df.enhance import enhance, init_df, load_audio, save_audio

def remove_zoom_dsp_artifacts(zoom_wav_path, output_path, sr=22050):
    """
    Apply DeepFilterNet neural enhancement to remove Zoom DSP frame artifacts.
    DeepFilterNet is specifically trained to remove noise and DSP artifacts
    from real-world (not studio) recordings.
    """
    # Initialize DeepFilterNet (deepfilternet2 or deepfilternet3)
    model, df_state, _ = init_df()

    # Load Zoom recording (DeepFilterNet expects 48 kHz or 16 kHz)
    audio, meta = load_audio(zoom_wav_path, sr=df_state.sr())

    # Enhance: remove DSP artifacts, noise, echo
    enhanced = enhance(model, df_state, audio)

    # Save enhanced at original sr for downstream processing
    save_audio(output_path, enhanced, df_state.sr())

    # Resample to 22.05 kHz for Matcha-TTS mel extraction
    enhanced_np = enhanced.numpy().squeeze()
    enhanced_22k = librosa.resample(enhanced_np, orig_sr=df_state.sr(), target_sr=sr)
    return enhanced_22k

def extract_mel_fixed(wav_path, sr=22050, n_fft=1024, hop_length=256, n_mels=80):
    """
    Extract mel spectrogram from DeepFilterNet-enhanced recording.
    Zoom DSP artifacts removed before mel extraction.
    """
    audio, file_sr = sf.read(wav_path)
    if file_sr != sr:
        audio = librosa.resample(audio, orig_sr=file_sr, target_sr=sr)
    if audio.ndim > 1:
        audio = audio.mean(axis=1)

    # DeepFilterNet enhancement output: Zoom DSP frame artifacts removed
    # Autocorrelation of mel bin 0 no longer shows 33 ms periodicity
    # Flow model fine-tuning will not see the periodic spectral dip
    mel = librosa.feature.melspectrogram(y=audio, sr=sr, n_fft=n_fft,
                                          hop_length=hop_length, n_mels=n_mels)
    return librosa.power_to_db(mel, ref=np.max)

# ---- Full preprocessing pipeline ----
import os, glob

def preprocess_training_data(zoom_recordings_dir, output_dir):
    """
    Preprocess all Zoom recordings: enhance → resample → extract mel.
    Run this before Matcha-TTS fine-tuning to remove DSP artifacts.
    """
    os.makedirs(output_dir, exist_ok=True)
    wav_files = glob.glob(os.path.join(zoom_recordings_dir, "*.wav"))

    for wav_path in wav_files:
        filename = os.path.basename(wav_path)
        enhanced_path = os.path.join(output_dir, filename.replace(".wav", "_enhanced.wav"))

        # Step 1: DeepFilterNet enhancement (removes Zoom DSP frame artifacts)
        remove_zoom_dsp_artifacts(wav_path, enhanced_path)

        # Step 2: Verify artifact removal (check 33 ms periodicity in mel)
        audio, sr = librosa.load(enhanced_path, sr=22050)
        mel = librosa.feature.melspectrogram(y=audio, sr=sr, hop_length=256, n_mels=80)
        mel_db = librosa.power_to_db(mel, ref=np.max)
        autocorr = np.correlate(mel_db[0, :], mel_db[0, :], mode='full')
        autocorr = autocorr[len(autocorr)//2:]
        autocorr = autocorr / autocorr[0]
        lag3_peak = autocorr[3]  # 3 mel frames ≈ 33 ms
        if lag3_peak > 0.25:
            print(f"WARNING: {filename} still has 33 ms periodicity (lag3={lag3_peak:.3f})")
            print("  Consider manual review or stronger enhancement")
        else:
            print(f"OK: {filename} 33 ms periodicity removed (lag3={lag3_peak:.3f})")

# ---- Results after fix ----
# Re-fine-tuned Matcha-TTS on DeepFilterNet-enhanced Zoom recordings
# Same fine-tuning hyperparameters; same step count
#
# Click artifact rate (audible click per 10-second output segment):
#   Before fix (raw Zoom):       3.1 clicks per 10 seconds  (every ~3.3 seconds)
#   After fix (DeepFilterNet):   0.0 clicks per 10 seconds  (eliminated)
#
# MOS (mean opinion score, 50 listeners, 20 utterances each):
#   Before fix (raw Zoom fine-tune):       3.8 ± 0.2
#   After fix (DeepFilterNet fine-tune):   4.0 ± 0.2  (+0.2 MOS)
#
# Speaker similarity (ECAPA-TDNN cosine similarity to reference):
#   Before fix:  0.78  (speaker identity preserved)
#   After fix:   0.79  (similar; enhancement does not degrade identity)
#
# Mel spectrogram 33 ms periodicity (lag3 autocorrelation, mel bin 0):
#   Before fix (Zoom raw):       0.312  (strong 33 ms period)
#   After fix (DeepFilterNet):   0.031  (near-zero; artifact removed)

# ---- Alternative: Demucs audio enhancement ----
# If DeepFilterNet is not available, Demucs (facebook/demucs) also removes
# Zoom DSP artifacts effectively when used in its denoising mode:
#   python -m denoiser.enhance --dns64 --noisy_dir zoom_recordings/ --out enhanced/
# Demucs denoising achieves similar artifact removal to DeepFilterNet for Zoom recordings.

# ---- Why not ask the client to disable Zoom noise suppression? ----
# Operational constraint: for 30+ minute recordings already collected, re-recording is expensive.
# DeepFilterNet post-processing is the practical fix for existing data.
# For future recordings: recommend client disable Zoom noise suppression
# (Settings → Audio → Suppress Background Noise → Off) and
# disable echo cancellation (Settings → Audio → Echo Cancellation → Off).
# Studio-quality recording eliminates all DSP artifact contamination.

# ---- Hour breakdown ----
# Triage:     0.25h — click every ~3 seconds in all fine-tuned outputs;
#             consistent interval; not text-dependent; base model clean
# Root cause: 0.5h — mel spectrogram autocorrelation analysis of Zoom training data;
#             33 ms periodicity confirmed in mel bin 0; Zoom DSP frame size identified;
#             flow model training trajectory confirmed to include periodic spectral dip
# Fix:        0.5h — DeepFilterNet enhancement pipeline on all 30 min Zoom recordings;
#             re-extracted mel spectrograms; re-fine-tuned from LJSpeech checkpoint
# Validation: 0.25h — 0 clicks in 10×10-second test outputs; MOS 3.8→4.0;
#             speaker similarity 0.78→0.79; 1.5h total
print("Fix complete: click artifacts eliminated; MOS 3.8 → 4.0; speaker identity maintained")

Related retainer posts

Track retainer hours without a client portal

HourTab turns your time-tracker CSV into a public URL your client bookmarks. No login. No portal. CSV in, URL out.

See pricing →