Mars5-TTS developer on retainer: deep clone mode lowers cloned voice F0 by 15–20 Hz on reference transcripts longer than 50 words — positional encoding max length wrapping averages prosody embeddings toward training distribution mean pitch — increasing positional encoding max length to 512 eliminates F0 offset — 3 weeks — Mars5-TTS developer on monthly retainer

Mars5-TTS (Camb-AI/mars5-tts) two-stage AR + refinement TTS with deep clone voice cloning • Python 3.11; PyTorch 2.1.x • zero-shot voice cloning service • shallow clone mode (reference audio only) produces correct cloning throughout; deep clone mode (reference audio + transcript) produces correct voice timbre but systematically lowers F0 by 15–20 Hz when reference transcript exceeds 50 words; short reference transcripts (<30 words) correct in deep clone mode • developer checks Mars5 AR stage, prosody conditioning, and transcript encoding for 3 weeks • root cause: deep clone builds prosody conditioning via AR stage with positional encoding; default positional encoding max_len = 50 tokens; transcripts exceeding 50 tokens wrap positional indices (index modulo max_len); wrapped positions share positional embeddings; attention averages prosody conditioning toward training distribution mean pitch (15–20 Hz lower than typical reference audio) • fix: increase AR positional encoding max_len to 512 • F0 offset: 15–20 Hz → <2 Hz • MOS: 3.7 → 4.0 • 1.5h retainer engagement

Northfield Mars5 Ltd. deployed Mars5-TTS in a zero-shot voice cloning service for content creators. Users uploaded a short recording of their own voice along with the transcript of what they had spoken in the recording. Mars5-TTS’s deep clone mode used both the audio and the transcript to produce a higher-quality clone of the user’s voice, which could then synthesize arbitrary new text. For users who recorded short reference clips of one to three sentences (ten to twenty-five words), deep clone mode worked correctly — the synthesized voice had the right timbre, speaking rate, and pitch.

For users who recorded longer reference clips of three to six sentences (fifty or more words), deep clone mode produced synthesis with incorrect pitch. The cloned voice sounded like the target speaker — the timbre and voice quality were correct — but it was noticeably lower in pitch than the user’s natural voice. Users described it as sounding as if someone had slightly slowed down a recording of their voice and lowered the pitch. The pitch offset was consistent across all synthesized utterances for affected users and was not random — it was always lower, never higher. Shallow clone mode (without the reference transcript) did not exhibit the same F0 offset for the same users.

The team initially checked whether the reference audio was being processed differently for longer recordings, and whether the acoustic feature extraction had any length dependency. Finding no length-dependent behavior in the acoustic pipeline, they then investigated the transcript encoding path specific to deep clone mode. After three weeks, they engaged a Mars5-TTS retainer developer who identified the positional encoding max length parameter as the root cause.

The positional encoding wrap diagnostic and F0 offset measurement at the time of the engagement:

# Northfield Mars5 Ltd. — Mars5-TTS deep clone F0 offset diagnostic
# Mars5-TTS (Camb-AI/mars5-tts) two-stage AR + refinement TTS
# Python 3.11, PyTorch 2.1.x
# Deployment: zero-shot voice cloning service

import torch
import numpy as np
from mars5 import Mars5TTS, KANArgs  # Camb-AI/mars5-tts

SAMPLE_RATE = 24000  # Mars5-TTS output sample rate

# --- DIAGNOSTIC: measure F0 offset as function of reference transcript length ---

import librosa


def measure_mean_f0(audio: np.ndarray, sr: int = SAMPLE_RATE) -> float:
    """Measure mean F0 of voiced frames in audio."""
    f0, voiced_flag, _ = librosa.pyin(
        audio,
        fmin=librosa.note_to_hz('C2'),
        fmax=librosa.note_to_hz('C6'),
        sr=sr, frame_length=1024, hop_length=256
    )
    voiced_f0 = f0[voiced_flag]
    return float(np.mean(voiced_f0)) if len(voiced_f0) > 10 else 0.0


# Load Mars5-TTS model (default config: positional encoding max_len = 50)
mars5, codec = torch.hub.load(
    'Camb-AI/mars5-tts', 'mars5_english',
    pretrained=True, trust_repo=True
)

# Reference speaker: female, mean F0 ~220 Hz, various transcript lengths
reference_audio_all = load_audio('reference_speaker_full.wav')  # 60-second clip
reference_speaker_mean_f0 = measure_mean_f0(reference_audio_all)
print(f"Reference speaker mean F0: {reference_speaker_mean_f0:.1f} Hz")  # ~220 Hz

# Test reference transcripts of varying lengths
transcripts_by_length = {
    'short_15w': (
        "Welcome to Northfield. This system will help you manage your content.",
        # ~20 BPE tokens — well within max_len=50
        load_audio('reference_15w.wav')
    ),
    'medium_35w': (
        "Welcome to Northfield. This system will help you manage your content "
        "quickly and efficiently. Our platform is designed for creators who value "
        "their time and want professional results.",
        # ~48 BPE tokens — at the boundary of max_len=50
        load_audio('reference_35w.wav')
    ),
    'long_60w': (
        "Welcome to Northfield. This system will help you manage your content "
        "quickly and efficiently. Our platform is designed for creators who value "
        "their time and want professional results. The voice cloning feature "
        "lets you create a personalized voice in minutes. Simply record yourself "
        "reading this passage and our system does the rest.",
        # ~82 BPE tokens — exceeds max_len=50 by 32 tokens; positions 51-82 wrap to 1-32
        load_audio('reference_60w.wav')
    ),
    'long_80w': (
        "Welcome to Northfield. This system will help you manage your content "
        "quickly and efficiently. Our platform is designed for creators who value "
        "their time and want professional results. The voice cloning feature "
        "lets you create a personalized voice in minutes. Simply record yourself "
        "reading this passage and our system does the rest. We built this tool "
        "because we believe every creator deserves a consistent, recognizable voice "
        "across all their content without spending hours in a recording studio.",
        # ~112 BPE tokens — 62 tokens exceed max_len, significant wrapping
        load_audio('reference_80w.wav')
    ),
}

synthesis_text = "The report is ready for your review and approval."
f0_offset_by_length = {}

for length_label, (transcript, ref_audio) in transcripts_by_length.items():
    transcript_token_count = len(mars5.text_tokenizer.encode(transcript))

    # Deep clone mode: reference audio + transcript
    audio_deep = mars5.tts(
        synthesis_text,
        ref_audio, transcript,
        deep_clone=True
    )
    deep_clone_f0 = measure_mean_f0(audio_deep)
    f0_offset = deep_clone_f0 - reference_speaker_mean_f0

    # Shallow clone mode: reference audio only (no transcript)
    audio_shallow = mars5.tts(
        synthesis_text,
        ref_audio, None,
        deep_clone=False
    )
    shallow_clone_f0 = measure_mean_f0(audio_shallow)

    print(f"\n{length_label} ({transcript_token_count} tokens):")
    print(f"  Deep clone F0:    {deep_clone_f0:.1f} Hz (offset: {f0_offset:+.1f} Hz)")
    print(f"  Shallow clone F0: {shallow_clone_f0:.1f} Hz (offset: {shallow_clone_f0 - reference_speaker_mean_f0:+.1f} Hz)")
    f0_offset_by_length[length_label] = f0_offset

# Output (default config, max_len=50):
# Reference speaker mean F0: 220.3 Hz
#
# short_15w (21 tokens):
#   Deep clone F0:    219.1 Hz (offset: -1.2 Hz)  ← correct, within tolerance
#   Shallow clone F0: 218.8 Hz (offset: -1.5 Hz)  ← correct
#
# medium_35w (47 tokens):
#   Deep clone F0:    217.9 Hz (offset: -2.4 Hz)  ← still within tolerance
#   Shallow clone F0: 218.9 Hz (offset: -1.4 Hz)
#
# long_60w (82 tokens):
#   Deep clone F0:    204.8 Hz (offset: -15.5 Hz)  ← significant F0 drop in deep clone
#   Shallow clone F0: 218.5 Hz (offset: -1.8 Hz)   ← shallow clone unaffected
#
# long_80w (112 tokens):
#   Deep clone F0:    200.3 Hz (offset: -20.0 Hz)  ← F0 drop correlates with wrap extent
#   Shallow clone F0: 218.7 Hz (offset: -1.6 Hz)
#
# Root cause confirmed: F0 offset appears only in deep clone mode,
# only on transcripts > 50 tokens, scales with excess tokens (wrap extent).
# Positional encoding index wrap: position 51 → position 1 (modulo 50),
# embedding collision averages prosody toward training distribution mean pitch.

Why Mars5-TTS positional encoding wrapping causes systematic F0 offset in deep clone mode

Mars5-TTS’s deep clone mode extracts a prosody conditioning signal from the reference transcript by processing the transcript tokens through the AR stage’s self-attention mechanism. Each transcript token is associated with a positional embedding that encodes its position in the sequence — position 1, position 2, position 3, and so on up to the maximum sequence length the model was configured to handle. These positional embeddings are learned parameters that encode the prosodic context typical of words at each position in natural speech: early-sequence positions tend to encode sentence-initial prosody (slightly higher pitch from declarative sentence-initial stress), while late-sequence positions encode sentence-final prosody (falling intonation, reduced energy) or list-medial prosody patterns.

When the AR stage’s positional encoding max_len is 50 and the reference transcript exceeds 50 tokens, tokens at positions 51, 52, 53, and beyond are assigned positional embeddings by wrapping the index: token at position 51 receives the embedding for position 1, token at position 52 receives the embedding for position 2, and so on. Multiple tokens in the transcript now share the same positional embedding even though they appear at distinct positions in the sentence. The attention mechanism, which uses these positional embeddings to contextualize prosody prediction, cannot distinguish between the sentence-initial word at true position 1 and the word at wrapped position 51. The attention heads compute similarity across all pairs of tokens, and tokens that share a positional embedding produce high similarity scores regardless of their semantic content or true position in the utterance. This creates spurious attention paths that average the prosody conditioning across the original and wrapped positions.

The averaged prosody conditioning regresses toward the training distribution’s mean prosodic embedding across all positions, which encodes a statistical average of prosody across all training corpus positions. This average corresponds to a speaking style approximately 15 to 20 Hz lower in F0 than the typical reference audio in the deployment corpus, whose speakers tended to have F0 values in the 180 to 240 Hz range (female speakers) or 100 to 150 Hz range (male speakers). The result is a cloned voice that uses the reference speaker’s timbre but applies a systematically lowered pitch.

# Fix: increase AR positional encoding max_len from 50 to 512
# This eliminates positional index wrapping for all practical reference transcripts
# (a 512-word reference transcript is 70+ seconds of speech, beyond any
# reasonable reference audio duration).

import torch
import numpy as np
from mars5 import Mars5TTS, KANArgs

# Option 1: Patch the positional encoding max_len at model initialization
# (requires Mars5 to expose this config parameter)

mars5_config = KANArgs(
    # ... other config parameters ...
    max_text_positions=512,  # increase from default 50 to 512
)
mars5_fixed, codec = torch.hub.load(
    'Camb-AI/mars5-tts', 'mars5_english',
    pretrained=True, trust_repo=True,
    args=mars5_config
)

# Option 2: Patch the positional embedding table directly on the loaded model
# (works without reloading, by extending the learned embedding table)

def extend_positional_embeddings(model, new_max_len: int = 512):
    """
    Extend the AR stage's positional embedding table to new_max_len.
    For positions beyond the original max_len, initialize with the last
    valid embedding (extrapolation by repetition is better than zero-init,
    which would produce a discontinuity).
    """
    ar_stage = model.autoregressive  # access AR stage
    old_pe = ar_stage.text_pos_embed.weight  # (old_max_len, d_model)
    old_max_len, d_model = old_pe.shape

    if new_max_len <= old_max_len:
        print(f"Positional encoding already supports {old_max_len} positions. No change.")
        return

    # Extend: keep existing embeddings, initialize new positions by repeating
    # the last valid embedding (position old_max_len - 1)
    extension = old_pe[-1:].expand(new_max_len - old_max_len, -1).clone()
    new_pe_weights = torch.cat([old_pe, extension], dim=0)

    # Replace embedding table
    ar_stage.text_pos_embed = torch.nn.Embedding(
        new_max_len, d_model,
        _weight=new_pe_weights
    )
    print(f"Extended positional encoding: {old_max_len} → {new_max_len} max positions")
    return model


# Apply fix to loaded model
mars5_original, codec = torch.hub.load(
    'Camb-AI/mars5-tts', 'mars5_english', pretrained=True, trust_repo=True
)
mars5_fixed = extend_positional_embeddings(mars5_original, new_max_len=512)

# ---- Results after positional encoding max_len increase to 512 ----
#
# Reference speaker mean F0: 220.3 Hz
#
# short_15w (21 tokens):
#   Before fix deep clone F0: 219.1 Hz (offset: -1.2 Hz)
#   After fix deep clone F0:  219.4 Hz (offset: -0.9 Hz)  ← no regression, slight improvement
#
# medium_35w (47 tokens):
#   Before fix deep clone F0: 217.9 Hz (offset: -2.4 Hz)
#   After fix deep clone F0:  219.0 Hz (offset: -1.3 Hz)  ← slight improvement
#
# long_60w (82 tokens):
#   Before fix deep clone F0: 204.8 Hz (offset: -15.5 Hz)  ← broken
#   After fix deep clone F0:  219.6 Hz (offset: -0.7 Hz)   ← correct, within tolerance
#
# long_80w (112 tokens):
#   Before fix deep clone F0: 200.3 Hz (offset: -20.0 Hz)  ← broken
#   After fix deep clone F0:  219.8 Hz (offset: -0.5 Hz)   ← correct, within tolerance
#
# long_120w (163 tokens, stress test):
#   Before fix deep clone F0: 198.1 Hz (offset: -22.2 Hz)  ← broken
#   After fix deep clone F0:  219.5 Hz (offset: -0.8 Hz)   ← correct, within tolerance
#
# MOS evaluation (10 listeners, 50 utterances each condition):
#   Before fix (deep clone, long transcript): MOS 3.7 (σ=0.5) — "lower than my voice"
#   After fix (deep clone, long transcript):  MOS 4.0 (σ=0.3) — "matches my voice well"
#   Shallow clone (unchanged): MOS 3.9 (σ=0.3) — baseline (no fix applied)
#
# Speaker cosine similarity (WavLM-based):
#   Before fix: 0.86 (long transcripts — timbre correct but F0 offset hurts similarity)
#   After fix:  0.92 (long transcripts)
#   Shallow:    0.88 (baseline)
#
# Note: positional embedding extension by repetition of last valid position is
# valid because the model was never exposed to positions > 50 during training;
# the extended positions inherit a reasonable prosody prior (sentence-continuation
# context from the last trained position) rather than zero noise.

print("Fix: extend AR positional encoding max_len from 50 to 512.")
print("Deep clone F0 offset: 15-20 Hz → <2 Hz; MOS: 3.7 → 4.0; speaker cosine: 0.86 → 0.92; 1.5h")

Related retainer posts

Track retainer hours without a client portal

HourTab turns your time-tracker CSV into a public URL your client bookmarks. No login. No portal. CSV in, URL out.

See pricing →