CosyVoice developer on retainer: fourth tone starting pitch 10–15 Hz lower than expected in cross-lingual Mandarin synthesis — English reference audio pitch IQR narrower than Mandarin norm — CosyVoice maps tonal contours into reference pitch ceiling — 3 weeks — CosyVoice developer on monthly retainer

CosyVoice (FunAudioLLM/CosyVoice) flow-matching TTS with supervised speech language model • Python 3.11; PyTorch 2.1.x • Mandarin Chinese e-learning platform • cross-lingual mode: English speaker voice for Mandarin text • fourth tone (falling, e.g., shì dà) starting pitch 10–15 Hz lower than first, second, third tones • developer checks CosyVoice cross-lingual parameters, reference audio pitch range, CosyVoice model version for 3 weeks • root cause: English read speech pitch IQR ~32 Hz vs Mandarin read speech ~70 Hz; CosyVoice speaker embedding encodes English pitch ceiling; fourth tone (high start, sharp fall) starting pitch must begin at pitch ceiling; English ceiling 12 Hz short of Mandarin norm • fix: PSOLA pitch range normalization to 70 Hz IQR on English reference before embedding extraction • fourth-tone starting pitch: +12 Hz • native Mandarin listener naturalness preference: 41% → 83% • 1.5h retainer engagement

Westfield CosyVoice TTS Ltd. built a Mandarin Chinese e-learning platform that uses CosyVoice cross-lingual synthesis to deliver lesson narration in a trusted English-speaking instructor’s voice. The cross-lingual mode extracts a speaker embedding from the English reference audio and applies it while synthesizing Mandarin text, producing Mandarin speech that sounds like it comes from the same person as the English recordings. For the first three Mandarin tones — the high level first tone, the rising second tone, and the falling-rising third tone — the synthesis sounded natural. But every occurrence of the fourth tone (the high-to-low falling tone used in words like 是 shì, 大 dà, 是 shì, 对 duì, 看 kàn) produced a starting pitch that was 10–15 Hz below what native Mandarin listeners expected. The fourth tone sounded unnaturally low relative to the other three tones, as if it were beginning its fall from a lower starting point than the speech register required.

The team checked the CosyVoice cross-lingual inference parameters, tried different English reference audio recordings (different speakers, different recording lengths from 5 to 30 seconds), updated to the latest CosyVoice model checkpoint, and consulted the CosyVoice documentation. All fourth tone outputs remained 10–15 Hz low regardless of which English reference was used. The pattern held across male and female English speakers, across different recording environments, and across different Mandarin text inputs. After three weeks, they engaged a CosyVoice retainer developer who identified the pitch IQR mismatch between English and Mandarin read speech as the structural cause, and built a PSOLA pitch normalization step that corrects the reference audio before embedding extraction.

The CosyVoice cross-lingual inference pipeline at the time of the engagement:

# Westfield CosyVoice TTS Ltd. — Mandarin e-learning cross-lingual synthesis
# CosyVoice (FunAudioLLM/CosyVoice), Python 3.11, PyTorch 2.1.x
# Use case: English speaker voice for Mandarin text narration

import torchaudio
from cosyvoice.cli.cosyvoice import CosyVoice
from cosyvoice.utils.file_utils import load_wav

cosyvoice = CosyVoice('pretrained_models/CosyVoice-300M')

# English reference audio — male professional narrator, US English accent
# Recorded in studio, clean, 24 kHz, 10 seconds read passage
# Pitch IQR measured with Praat: approximately 32 Hz (English read speech norm)
reference_audio = load_wav('reference_en_male_narrator.wav', 24000)

def synthesize_mandarin_cross_lingual_buggy(text_zh, reference_audio):
    """
    Cross-lingual synthesis: produce Mandarin text in English speaker voice.

    CosyVoice cross-lingual mode:
    1. Extracts speaker embedding from reference_audio (English male narrator)
    2. Speaker embedding captures pitch range, timbre, rhythm from reference
    3. Synthesizes text_zh using the extracted speaker characteristics

    Bug: fourth tone (falling tone) starting pitch 10-15 Hz lower than expected.
    Fourth tone sounds unnaturally low relative to first/second/third tones.

    Example fourth tone pitch measurements (shì, 是):
      With English reference:  183 Hz starting pitch (observed)
      Mandarin native norm:    195 Hz starting pitch (expected for male voice)
      Shortfall:               12 Hz lower than Mandarin norm

    First/second/third tone measurements — close to norm (within 5 Hz)
    """
    output = cosyvoice.inference_cross_lingual(
        tts_text=text_zh,
        prompt_speech_16k=reference_audio,
    )
    return output['tts_speech']

# Root cause analysis:
# CosyVoice extracts a speaker embedding that encodes pitch dynamic range.
# English read speech pitch IQR (interquartile range, 25th-75th percentile):
#   Male US narrator:     ~32 Hz (P25=148 Hz, P50=162 Hz, P75=180 Hz)
#   Female US narrator:   ~40 Hz (P25=182 Hz, P50=200 Hz, P75=222 Hz)
#
# Mandarin read speech pitch IQR (required for distinct 4-tone lexical contrast):
#   Male Mandarin native: ~68 Hz (P25=118 Hz, P50=147 Hz, P75=186 Hz)
#   Female Mandarin native: ~75 Hz (P25=180 Hz, P50=215 Hz, P75=255 Hz)
#
# CosyVoice maps Mandarin tonal contours into the reference pitch range.
# For fourth tone (high-to-low fall, starting at pitch ceiling):
#   English male P75 = 180 Hz → fourth tone starting pitch clips at ~183 Hz
#   Mandarin male norm = 195 Hz → 12 Hz above English ceiling
#
# First tone (high, level): uses P75 of pitch range → also slightly low
#   but less perceptible because first tone is sustained, not a sharp fall
# Second tone (rising): uses P25 to P75 → lower range, not ceiling-limited
# Third tone (fall-rise dip): uses lower half of pitch range → not ceiling-limited
# Fourth tone (sharp fall from ceiling): most severely affected by narrow English IQR

print("Root cause: English pitch IQR ~32 Hz vs Mandarin ~70 Hz")
print("Fourth tone starts at English pitch ceiling — 12 Hz short of Mandarin norm")
print("First/second/third tones less affected: do not require absolute pitch ceiling")

Why English reference audio produces low fourth tone pitch in CosyVoice cross-lingual synthesis

Mandarin Chinese uses four lexical tones to distinguish word meaning: the first tone (high level, like mā 妈 “mother”), second tone (rising, má 麻 “hemp”), third tone (falling-rising dip, mă 马 “horse”), and fourth tone (high-to-low sharp fall, mà 骂 “scold”). Unlike English prosodic pitch, which varies pitch at the sentence level for emphasis and intonation, Mandarin tonal pitch must achieve specific absolute pitch heights at the syllable level to convey lexical meaning. This means Mandarin speakers use a wider pitch dynamic range per utterance than English speakers producing the same number of syllables. The empirical difference: Mandarin read speech pitch IQR is approximately 60–90 Hz; English read speech pitch IQR is approximately 30–50 Hz, nearly half as wide.

CosyVoice extracts a speaker embedding from the reference audio that captures pitch range characteristics alongside timbre and rhythm. When the reference audio comes from an English speaker, the embedding encodes a pitch ceiling of approximately 180 Hz (for a male narrator) or 220 Hz (female narrator). The fourth Mandarin tone must begin its fall at or near this ceiling. For a native Mandarin male speaker, the fourth tone starting pitch norms at approximately 195 Hz — 15 Hz above the English male narrator’s pitch ceiling. The fix is to apply PSOLA pitch manipulation to the English reference audio to stretch its pitch IQR from ~32 Hz to 70 Hz (matching Mandarin read speech) before the CosyVoice speaker embedding is extracted. PSOLA modifies the pitch contour by scaling each voiced pitch value away from the median pitch by the ratio of target-IQR to source-IQR, while preserving formant structure and speaking rate. After normalization, the English reference audio’s pitch ceiling is raised to approximately 196 Hz, allowing the CosyVoice fourth tone synthesis to reach the Mandarin norm.

# Fix: PSOLA pitch IQR normalization before CosyVoice embedding extraction
# Stretch English reference pitch IQR from ~32 Hz to 70 Hz (Mandarin norm)

import numpy as np
import parselmouth  # Praat Python bindings for PSOLA pitch manipulation
from parselmouth.praat import call
from cosyvoice.cli.cosyvoice import CosyVoice
from cosyvoice.utils.file_utils import load_wav

cosyvoice = CosyVoice('pretrained_models/CosyVoice-300M')

TARGET_PITCH_IQR_HZ = 70.0  # Mandarin read speech pitch IQR norm

def normalize_pitch_range_psola(reference_wav_path,
                                  target_iqr_hz=TARGET_PITCH_IQR_HZ):
    """
    Normalize pitch dynamic range to target IQR using PSOLA.

    Steps:
    1. Measure current voiced pitch IQR with Praat pitch analysis
    2. Compute IQR scale factor: target_iqr / current_iqr
    3. Scale each voiced pitch value around the median using PSOLA:
       new_pitch = median + (old_pitch - median) * scale_factor
    4. Resynthesize with scaled pitch contour (formants and duration unchanged)

    Preserves: speaker timbre (formant structure), speaking rate, voice identity
    Modifies: pitch dynamic range only (wider IQR, raised pitch ceiling)
    """
    sound = parselmouth.Sound(reference_wav_path)

    # Step 1: Measure current pitch IQR
    pitch = sound.to_pitch(time_step=0.01, pitch_floor=75, pitch_ceiling=600)
    pitch_values = pitch.selected_array['frequency']
    voiced = pitch_values[pitch_values > 0]

    p25 = np.percentile(voiced, 25)
    p50 = np.percentile(voiced, 50)   # Median (preserve this)
    p75 = np.percentile(voiced, 75)
    current_iqr = p75 - p25

    # Male US narrator: p50≈162 Hz, IQR≈32 Hz, p75≈180 Hz (measured)
    print(f"Source: median={p50:.1f} Hz, IQR={current_iqr:.1f} Hz "
          f"(P25={p25:.1f}, P75={p75:.1f})")

    scale_factor = target_iqr_hz / current_iqr
    # scale_factor ≈ 70 / 32 ≈ 2.19 for English male narrator

    # Step 2: PSOLA pitch manipulation via Praat Manipulation object
    manipulation = call(sound, "To Manipulation", 0.01, 75, 600)
    pitch_tier = call(manipulation, "Extract pitch tier")

    num_points = call(pitch_tier, "Get number of points")
    for i in range(1, num_points + 1):
        t = call(pitch_tier, "Get time from index", i)
        old_pitch = call(pitch_tier, "Get value at index", i)
        # Scale around median: raise pitch ceiling, lower pitch floor
        new_pitch = p50 + (old_pitch - p50) * scale_factor
        new_pitch = max(75.0, min(600.0, new_pitch))  # Praat valid range
        call(pitch_tier, "Remove point near", t, 0.001)
        call(pitch_tier, "Add point", t, new_pitch)

    # Step 3: Replace pitch tier and resynthesize
    call([pitch_tier, manipulation], "Replace pitch tier")
    normalized_sound = call(manipulation, "Get resynthesis (overlap-add)")

    output_path = reference_wav_path.replace('.wav', '_pitch_normalized.wav')
    normalized_sound.save(output_path, 'WAV')

    # Verify normalized IQR
    p_norm = normalized_sound.to_pitch(time_step=0.01, pitch_floor=75, pitch_ceiling=600)
    pv_norm = p_norm.selected_array['frequency']
    new_iqr = (np.percentile(pv_norm[pv_norm > 0], 75) -
               np.percentile(pv_norm[pv_norm > 0], 25))
    print(f"Normalized: median≈{p50:.1f} Hz, IQR={new_iqr:.1f} Hz "
          f"(target {target_iqr_hz:.1f} Hz)")
    # Expected: ~68-72 Hz (slight deviation due to PSOLA discretization)

    return output_path

def synthesize_mandarin_cross_lingual_fixed(text_zh, reference_wav_path):
    """
    Cross-lingual synthesis with pitch range normalization.
    Normalizes English reference pitch IQR to 70 Hz before embedding extraction.
    CosyVoice then maps fourth tone into the expanded pitch range correctly.
    """
    normalized_path = normalize_pitch_range_psola(reference_wav_path)
    normalized_audio = load_wav(normalized_path, 24000)

    output = cosyvoice.inference_cross_lingual(
        tts_text=text_zh,
        prompt_speech_16k=normalized_audio,
    )
    return output['tts_speech']

# ---- Results after PSOLA pitch IQR normalization ----
#
# Fourth tone (shì, 是) — starting pitch:
#   Before fix:  183 Hz (12 Hz below Mandarin male norm of 195 Hz)
#   After fix:   197 Hz (+14 Hz; 2 Hz above norm — slight PSOLA overshoot)
#
# Native Mandarin listener naturalness test (50 listeners, 4-AFC by tone):
#   1st tone (ā):  Before 76% natural; After 78% — marginal
#   2nd tone (á):  Before 74% natural; After 75% — marginal
#   3rd tone (ǎ):  Before 72% natural; After 74% — marginal
#   4th tone (à):  Before 41% natural; After 83% — primary fix target
#
# Overall cross-lingual naturalness preference (blind A/B test):
#   English reference, without normalization: 47% preferred over native speaker
#   English reference, with normalization:    83% preferred over native speaker
#   Native Mandarin reference (upper bound):  91% preferred
#
# Speaker identity cosine similarity (CosyVoice speaker embeddings):
#   Before normalization: 1.000 (baseline)
#   After normalization:  0.943 (pitch range widened; timbre preserved)
#   Acceptable threshold: > 0.90

# ---- Pitch IQR measurements (Praat, voiced frames only) ----
#   English reference, original:    IQR=32 Hz (P25=148, P50=162, P75=180 Hz)
#   English reference, normalized:  IQR=69 Hz (P25=131, P50=162, P75=200 Hz)
#   Mandarin male reference:        IQR=68 Hz (P25=118, P50=147, P75=186 Hz)
#   (Mandarin male is lower overall pitch; normalization is speaker-relative)

print("Fix: PSOLA IQR normalization; fourth-tone starting pitch +14 Hz")
print("Native Mandarin listener naturalness: 41% → 83%; speaker cosine 0.943")

Related retainer posts

Track retainer hours without a client portal

HourTab turns your time-tracker CSV into a public URL your client bookmarks. No login. No portal. CSV in, URL out.

See pricing →