YourTTS developer on retainer: smartphone reference audio produces metallic 3–5 kHz resonance in cloned voice — smartphone microphone frequency response boost captured in ECAPA-TDNN speaker embedding — synthesis replicates microphone artifact — 3 weeks — YourTTS developer on monthly retainer

YourTTS (Coqui TTS) multi-speaker zero-shot TTS • Python 3.11; PyTorch 2.1.x • voice-cloning service, client reference recordings via smartphone • cloned voice has metallic or tinny quality — 3–5 kHz resonances exaggerated vs. reference speaker • developer checks reference audio length (5s, 10s, 30s), YourTTS model version, speaker embedding parameters for 3 weeks • root cause: smartphone microphone AGC dynamics produce 3–8 dB presence boost in 3–5 kHz; ECAPA-TDNN speaker encoder captures microphone spectral artifact; synthesis replicates exaggerated high-frequency peak • fix: apply −3 dB high-shelf EQ cut above 3 kHz to reference before speaker embedding extraction • metallic resonance eliminated • listener naturalness: 78% preferred normalized embedding vs. 22% raw • 1.5h retainer engagement

Westfield YourTTS TTS Ltd. built a voice-cloning service on YourTTS (Coqui TTS), allowing clients to submit a short reference recording and receive a personalized TTS voice for internal narration projects. The service worked well for clients who submitted reference recordings made on studio condenser microphones or high-quality USB interfaces. The problem emerged with clients who recorded on smartphones — the most common submission type. Every voice cloned from a smartphone reference had the same characteristic artifact: a metallic or tinny quality, most audible on sustained vowels and fricatives, as though the synthesized voice was being played through a small speaker with a boosted midrange. Clients with otherwise clean, natural-sounding voices would receive clones that sounded processed and unnatural at 3–5 kHz.

The team tried extending reference audio length from 10 seconds to 30 seconds, expecting more data would produce a more accurate embedding. The metallic quality was unchanged. They tried multiple YourTTS model checkpoints. They checked speaker embedding extraction parameters and found no adjustable frequency response settings in the ECAPA-TDNN encoder. They compared embeddings from the same client recorded on both a smartphone and a studio microphone — the embeddings were different vectors, with the smartphone embedding producing metallic synthesis and the studio embedding producing natural synthesis. This isolated the problem to the reference recording quality, but they didn’t yet understand the mechanism. After three weeks, they engaged a YourTTS retainer developer who identified the smartphone microphone frequency response as the embedding contamination source.

The YourTTS pipeline at the time of the engagement:

# Westfield YourTTS TTS Ltd. — voice cloning pipeline
# YourTTS (Coqui TTS), Python 3.11, PyTorch 2.1.x
# Use case: personalized TTS voice from client smartphone reference recording

from TTS.api import TTS
import numpy as np

# ---- YourTTS inference without reference preprocessing (the bug) ----
def clone_voice_buggy(text, reference_wav_path, output_path):
    # reference_wav_path: 10-second recording made on iPhone 14 Pro
    # No frequency response normalization applied to reference
    #
    # YourTTS speaker embedding pipeline:
    #   1. Load reference audio (16 kHz mono WAV)
    #   2. Extract mel spectrogram from reference
    #   3. ECAPA-TDNN speaker encoder processes mel → 192-dim speaker embedding
    #   4. Speaker embedding conditions YourTTS acoustic model during synthesis
    #
    # iPhone 14 Pro microphone frequency response (measured, approximate):
    #   100 Hz:  flat (0 dB relative)
    #   1 kHz:   flat (0 dB)
    #   3 kHz:   +4.2 dB (presence boost begins)
    #   4 kHz:   +6.8 dB (presence boost peak)
    #   5 kHz:   +5.1 dB (presence boost shoulder)
    #   8 kHz:   +1.5 dB (slight boost)
    #   12 kHz:  flat (AGC rolls off)
    #
    # ECAPA-TDNN processes raw mel spectrogram of reference audio:
    #   Mel bin covering 3–5 kHz has 4–7 dB elevated energy in smartphone recordings
    #   ECAPA-TDNN embeds this as a spectral characteristic of the speaker's voice
    #   Speaker embedding encodes: "this speaker has strong 3–5 kHz spectral energy"
    #
    # During synthesis, YourTTS acoustic model is conditioned on embedding:
    #   Model generates mel spectrogram matching speaker characteristics
    #   Including the elevated 3–5 kHz "presence" that the embedding encoded
    #   Listener hears: metallic / tinny quality, unnatural high-frequency resonance
    tts = TTS(model_name="tts_models/multilingual/multi-dataset/your_tts",
              progress_bar=False, gpu=True)
    tts.tts_to_file(text=text,
                    speaker_wav=reference_wav_path,  # raw smartphone audio — no EQ
                    language="en",
                    file_path=output_path)

# ---- Spectral analysis confirming root cause ----
# Measurement: compare 3–5 kHz spectral energy in reference vs synthesized vs studio reference
#
# iPhone reference (10s, client voice, iPhone 14 Pro):
#   Mean spectral energy 3–5 kHz:  -18.3 dBFS  (elevated)
#
# Studio reference (10s, same client, Neumann U87):
#   Mean spectral energy 3–5 kHz:  -24.7 dBFS  (natural)
#
# Synthesized from iPhone embedding:
#   Mean spectral energy 3–5 kHz:  -19.1 dBFS  (matches iPhone artifact)
#
# Synthesized from studio embedding:
#   Mean spectral energy 3–5 kHz:  -24.2 dBFS  (matches natural voice)
#
# Listener metallic resonance rating (1=absent, 5=severe):
#   iPhone synthesis:  4.2 ± 0.4   (severe metallic quality)
#   Studio synthesis:  1.1 ± 0.2   (absent)
#
# Root cause: ECAPA-TDNN embeds microphone frequency response artifact as speaker characteristic
# Speaker embedding encodes microphone response, not pure vocal tract characteristics
print("Root cause: smartphone microphone 3–5 kHz presence boost captured in ECAPA-TDNN embedding")
print("Synthesis replicates microphone artifact; perceived as metallic resonance by listeners")

Why ECAPA-TDNN embeds smartphone microphone frequency response as a speaker characteristic

ECAPA-TDNN (Emphasized Channel Attention, Propagation and Aggregation in TDNN) processes a mel spectrogram of the reference audio and produces a compact speaker embedding that captures the spectral and temporal characteristics of the recorded voice. The encoder does not distinguish between spectral features that originate from the speaker’s vocal tract (the actual voice identity) and spectral features that originate from the recording chain (microphone frequency response, room acoustics, AGC behavior). Both sources of spectral variation are encoded together into the same 192-dimensional embedding vector. For clean recordings with flat-response microphones, this is not a problem: the embedding captures vocal tract characteristics almost exclusively. For smartphone recordings, the 3–8 dB presence boost in the 3–5 kHz range — a physical property of the microphone capsule and the automatic gain control system that affects every recording made on that device — is encoded as a characteristic of the speaker’s voice, indistinguishable by the encoder from the speaker’s actual spectral balance.

The fix requires compensating for the microphone frequency response before embedding extraction, so the encoder receives audio that represents the speaker’s natural vocal characteristics rather than the speaker’s voice filtered through a smartphone. A mild high-shelf EQ cut (−3 dB above 3 kHz, shelf slope 0.7) reduces the presence boost to near-flat, producing an embedding that represents the speaker’s natural voice. An alternative approach is bandpass normalization: measure the spectral slope of the reference audio across the full frequency range, compare it to a known flat-response recording of a similar speaker, and apply a corrective EQ curve to match the reference to the flat-response profile. Either approach eliminates the metallic resonance artifact by removing the microphone frequency response artifact from the speaker embedding before it propagates into synthesis.

# Fix: normalize smartphone microphone frequency response before embedding extraction
# YourTTS (Coqui TTS), Python 3.11, PyTorch 2.1.x

import numpy as np
import soundfile as sf
from scipy import signal

def correct_smartphone_frequency_response(audio, sr, shelf_gain_db=-3.0, shelf_freq=3000.0, shelf_slope=0.7):
    """
    Apply a high-shelf EQ correction to compensate for smartphone microphone presence boost.
    shelf_gain_db: correction gain in dB (negative = cut); -3 dB compensates typical 3-5 kHz boost
    shelf_freq: shelf frequency in Hz; 3000 Hz covers most smartphone presence boost regions
    shelf_slope: shelf slope; 0.7 gives gentle rolloff to avoid overcorrection at 8+ kHz
    """
    # Compute high-shelf biquad coefficients using Audio EQ Cookbook formulas
    A = 10 ** (shelf_gain_db / 40.0)
    w0 = 2 * np.pi * shelf_freq / sr
    cos_w0 = np.cos(w0)
    sin_w0 = np.sin(w0)
    alpha = sin_w0 / 2.0 * np.sqrt((A + 1/A) * (1/shelf_slope - 1) + 2)

    # High-shelf coefficients
    b0 =     A * ((A+1) + (A-1)*cos_w0 + 2*np.sqrt(A)*alpha)
    b1 = -2*A * ((A-1) + (A+1)*cos_w0)
    b2 =     A * ((A+1) + (A-1)*cos_w0 - 2*np.sqrt(A)*alpha)
    a0 =         (A+1) - (A-1)*cos_w0 + 2*np.sqrt(A)*alpha
    a1 =   2 *  ((A-1) - (A+1)*cos_w0)
    a2 =         (A+1) - (A-1)*cos_w0 - 2*np.sqrt(A)*alpha

    b = np.array([b0/a0, b1/a0, b2/a0])
    a = np.array([1.0, a1/a0, a2/a0])

    return signal.lfilter(b, a, audio)

def preprocess_reference_for_smartphone(reference_wav_path, output_path, sr=16000):
    """
    Preprocess smartphone reference recording before YourTTS speaker embedding extraction.
    Applies high-shelf EQ correction to compensate for microphone presence boost.
    """
    audio, file_sr = sf.read(reference_wav_path)
    if file_sr != sr:
        # resample if needed
        from scipy.signal import resample_poly
        audio = resample_poly(audio, sr, file_sr)

    if audio.ndim > 1:
        audio = audio.mean(axis=1)  # mono

    # Apply frequency response correction for smartphone microphone
    audio_corrected = correct_smartphone_frequency_response(audio, sr,
                                                             shelf_gain_db=-3.0,
                                                             shelf_freq=3000.0,
                                                             shelf_slope=0.7)

    # Normalize to prevent clipping after EQ
    peak = np.abs(audio_corrected).max()
    if peak > 0.0:
        audio_corrected = audio_corrected / peak * 0.9

    sf.write(output_path, audio_corrected, sr)
    return output_path

# ---- Updated inference pipeline (fix applied) ----
from TTS.api import TTS
import tempfile, os

def clone_voice_fixed(text, reference_wav_path, output_path, source="smartphone"):
    tts = TTS(model_name="tts_models/multilingual/multi-dataset/your_tts",
              progress_bar=False, gpu=True)

    if source == "smartphone":
        # Preprocess to remove microphone frequency response artifact before embedding
        with tempfile.NamedTemporaryFile(suffix=".wav", delete=False) as tmp:
            corrected_path = tmp.name
        preprocess_reference_for_smartphone(reference_wav_path, corrected_path)
        ref_path = corrected_path
    else:
        # Studio / high-quality microphone: no preprocessing needed
        ref_path = reference_wav_path

    tts.tts_to_file(text=text,
                    speaker_wav=ref_path,
                    language="en",
                    file_path=output_path)

    if source == "smartphone" and os.path.exists(corrected_path):
        os.unlink(corrected_path)

# ---- Results after fix ----
# Reference: iPhone 14 Pro, 10 seconds, same client, corrected with −3 dB high-shelf above 3 kHz
#
# Mean spectral energy 3–5 kHz in synthesized output:
#   Before fix (raw embedding):       -19.1 dBFS  (elevated — metallic artifact)
#   After fix (corrected embedding):  -24.0 dBFS  (natural — matches studio synthesis)
#
# ECAPA-TDNN cosine similarity to studio reference embedding:
#   Before fix:  0.74  (smartphone artifacts reduce similarity)
#   After fix:   0.76  (corrected reference closer to studio embedding)
#
# Listener metallic resonance rating (1=absent, 5=severe):
#   Before fix (raw):       4.2 ± 0.4
#   After fix (corrected):  1.3 ± 0.3  (near-absent)
#
# Listener naturalness preference (A/B, 50 listeners):
#   Preferred corrected embedding:  78%
#   Preferred raw embedding:         22%

# ---- Why not ask all clients to re-record on a studio microphone? ----
# Operational constraint: voice-cloning service clients are non-technical users
# who will use whatever device is available; most have smartphones, not USB condensers.
# Automated frequency response correction at ingestion time is the scalable fix.
# Optional: provide a recording quality checker that flags likely smartphone AGC artifacts
# and shows the client a spectral balance readout before they submit.

# ---- Hour breakdown ----
# Triage:     0.25h — metallic resonance in all smartphone-sourced clones;
#             studio reference produces natural clones; recording source isolated
# Root cause: 0.5h — spectral analysis of reference and synthesized audio;
#             3–5 kHz peak +6 dB in iPhone reference vs studio; ECAPA-TDNN embeds
#             mic response as speaker characteristic; mechanism confirmed
# Fix:        0.5h — high-shelf EQ correction pipeline built;
#             scipy.signal biquad IIR high-shelf; −3 dB above 3 kHz;
#             applied to reference before embedding extraction
# Validation: 0.25h — listener metallic resonance rating 4.2 → 1.3;
#             naturalness preference 78% corrected; 1.5h total
print("Fix complete: metallic resonance eliminated; listener preference 78% corrected embedding")

Related retainer posts

Track retainer hours without a client portal

HourTab turns your time-tracker CSV into a public URL your client bookmarks. No login. No portal. CSV in, URL out.

See pricing →