HierSpeech++ developer on retainer: pitch mean shifted 15–20 Hz higher than target reference speaker — rising-intonation reference audio biases style encoder pitch representation upward — normalizing flow conditions synthesis on biased pitch — 3 weeks — HierSpeech++ developer on monthly retainer

HierSpeech++ (sh-lee-github/hierspeech) hierarchical variational autoencoder TTS with normalizing flow • Python 3.11; PyTorch 2.1.x • voice conversion service • 5-second reference clip per target speaker • all converted speech: pitch mean 15–20 Hz higher than reference speaker’s natural neutral pitch mean • speaker identity cosine similarity remains high (0.91–0.94) • offset consistent across all source speakers and all target texts • developer checks HierSpeech++ configuration, normalizing flow parameters, speaker embedding pipeline for 3 weeks • root cause: HierSpeech++ speech style encoder extracts style vector capturing pitch-level characteristics from the reference clip; reference clips containing rising-intonation sentences (questions) bias the pitch representation 15–20 Hz above the speaker’s neutral declarative mean; normalizing flow conditions synthesis on this biased style vector • fix: replace 5-second reference clips with 10+ seconds of declarative reference audio (no questions or exclamations); measure reference pitch mean against speaker’s neutral mean; require within ±3 Hz before accepting reference • pitch offset: 15–20 Hz → <2 Hz • MOS: 3.8 → 4.0 • 1.5h retainer engagement

Westfield HierSpeech TTS Ltd. built a voice conversion service using HierSpeech++ for a podcast production workflow, converting source speakers’ voices to target speaker profiles without changing the content. The system was used by a production company to convert recorded interviews to a consistent set of studio-quality voice profiles. Voice identity similarity between the converted output and the target speaker was high, confirmed by cosine similarity measurements above 0.91 across all tested target speakers. However, the production team noticed a systematic quality issue in the converted audio: every target speaker’s converted output sounded slightly higher in pitch than their natural speaking voice, as if each speaker had been subtly shifted into a higher register. The pitch mean measured over 50-utterance test sets was consistently 15–20 Hz above the target speaker’s natural neutral pitch mean, regardless of the source speaker being converted.

The team varied the conversion parameters: different HierSpeech++ temperature settings, different normalizing flow inference configurations, different speaker embedding extraction methods. The pitch offset persisted uniformly across all configurations. They checked whether the target speaker embeddings themselves were biased, running direct synthesis (non-conversion) with each target speaker embedding — the direct synthesis outputs had correct pitch. The offset was specific to the voice conversion pathway and appeared consistently only when the HierSpeech++ style encoder was involved. After three weeks without resolution, they engaged a HierSpeech++ retainer developer who identified that the reference audio clips used for style extraction all contained at least one question sentence, and the rising intonation of those questions was biasing the style encoder’s pitch representation upward by 15–20 Hz.

The HierSpeech++ voice conversion pipeline at the time of the engagement:

# Westfield HierSpeech TTS Ltd. — voice conversion pipeline
# HierSpeech++ (sh-lee-github/hierspeech), Python 3.11, PyTorch 2.1.x
# Use case: convert recorded interview audio to consistent studio voice profiles

import torch
import torchaudio
import numpy as np
from hierspeech import HierSpeechPP
from hierspeech.style_encoder import StyleEncoder

model = HierSpeechPP.from_pretrained("sh-lee-github/hierspeech-pp-large")
style_encoder = StyleEncoder.from_pretrained("sh-lee-github/hierspeech-pp-large")

def extract_style_vector(reference_path):
    """
    Extract style vector from reference audio for HierSpeech++ conditioning.

    HierSpeech++ architecture (relevant to the bug):
    1. StyleEncoder extracts a style vector s from the reference clip
       s captures: speaker identity (formant, spectral envelope, voice quality)
                   AND pitch-level characteristics (fundamental frequency mean, range)
    2. The normalizing flow f_theta maps source speech to target style:
       converted = f_theta(source_features, s)
    3. Pitch level in converted output is modulated by s's pitch representation

    Bug: reference clips were recorded as short introductory phrases including
    at least one rising-intonation question sentence. The style encoder encodes
    the rising F0 trajectory of the question into the style vector's pitch
    representation, biasing the extracted pitch mean 15–20 Hz above the
    speaker's neutral declarative pitch mean.
    """
    wav, sr = torchaudio.load(reference_path)
    if sr != 22050:
        wav = torchaudio.functional.resample(wav, sr, 22050)

    # Style encoder processes reference audio, extracting pitch-sensitive features
    # Bug: if reference contains questions (rising F0), pitch representation is biased
    with torch.no_grad():
        style_vector = style_encoder.encode(wav)

    # style_vector shape: [1, style_dim] — encodes speaker + pitch characteristics
    # Pitch representation in style_vector is biased if reference has rising intonation
    return style_vector  # Bug: upward pitch bias from rising-intonation reference

def convert_voice_buggy(source_path, reference_path):
    """
    Convert source speech to target speaker voice.
    Bug: systematic +15–20 Hz pitch offset in converted output.
    """
    source_wav, sr = torchaudio.load(source_path)
    if sr != 22050:
        source_wav = torchaudio.functional.resample(source_wav, sr, 22050)

    style_vector = extract_style_vector(reference_path)

    # Normalizing flow conditions on the biased style vector
    # Converted output pitch mean follows the biased pitch representation
    with torch.no_grad():
        converted = model.convert(
            source=source_wav,
            style=style_vector,  # Pitch-biased from rising-intonation reference
        )

    return converted

# Diagnosis: pitch offset measurement across 10 target speakers
#
# Reference clip content (5 seconds each):
#   "Hi, I'm [name]. Can you hear me okay? Great, let's get started."
#   Structure: greeting (declarative) + question (rising) + response (declarative)
#
# Measured pitch means (F0 extracted with CREPE, over 50-utterance test set):
#
#   Speaker 1: neutral mean 118 Hz | reference clip mean 134 Hz | converted mean 133 Hz
#              Offset: +15 Hz (reference biased by rising-intonation question)
#
#   Speaker 2: neutral mean 142 Hz | reference clip mean 161 Hz | converted mean 160 Hz
#              Offset: +18 Hz
#
#   Speaker 3: neutral mean 97  Hz | reference clip mean 115 Hz | converted mean 114 Hz
#              Offset: +17 Hz
#
#   Speaker 4: neutral mean 203 Hz | reference clip mean 221 Hz | converted mean 220 Hz
#              Offset: +17 Hz
#
#   Speaker 5: neutral mean 178 Hz | reference clip mean 196 Hz | converted mean 195 Hz
#              Offset: +17 Hz
#
# Pattern: reference clip pitch mean systematically 15–20 Hz above speaker neutral mean
#          Converted output pitch mean tracks reference clip mean, not neutral mean
#          Rising-intonation question ("Can you hear me okay?") pushes clip mean up
#
# Control: direct synthesis with same target speaker embedding (no conversion):
#   Pitch mean within ±2 Hz of neutral mean → pitch bias is in style encoder pathway

print("Root cause: rising-intonation question in reference clip biases style encoder pitch")
print("Style encoder pitch representation: neutral mean + 15–20 Hz from rising intonation")
print("Normalizing flow conditions synthesis on biased pitch → converted output +15–20 Hz")

Why HierSpeech++ style encoder is sensitive to reference audio intonation

HierSpeech++ uses a hierarchical architecture where a speech style encoder extracts a compact style representation from the reference audio. This style representation is designed to capture the perceptual characteristics of the target speaker’s voice so that the normalizing flow can map any source speaker’s content features to the target style. The style encoder must capture pitch-related characteristics because pitch is fundamental to perceived voice identity — a speaker who always uses a higher pitch register sounds different from the same speaker at a lower register, even if all other phonetic features are identical. The style encoder therefore encodes the pitch level of the reference audio as part of the extracted style vector.

The problem arises because pitch level in a natural reference audio clip is a mixture of two components: the speaker’s baseline fundamental frequency (a stable individual characteristic) and the intonation pattern of the specific sentences in the clip (variable, determined by sentence type and discourse context). Rising-intonation questions in English characteristically raise the fundamental frequency by 15–30 Hz in the final portion of the utterance, moving from the sentence’s mid-range up toward the speaker’s maximum comfortable pitch. A 5-second reference clip containing one rising-intonation question sentence has a measured pitch mean that is 15–20 Hz above the speaker’s declarative neutral mean, because the rising portion of the question (the final 1–2 seconds) raises the average. The style encoder extracts this elevated pitch mean as part of the style vector, indistinguishable from a speaker who naturally has a 15–20 Hz higher baseline pitch.

The fix is straightforward once the root cause is understood: select reference audio that consists entirely of declarative sentences — statements, descriptions, or explanations — where sentence-final intonation is neutral or slightly falling rather than rising. With 10 or more seconds of declarative speech, the measured pitch mean of the reference clip falls within ±3 Hz of the speaker’s actual neutral mean, and the style vector extracted from this reference encodes the speaker’s true pitch baseline rather than an intonation artifact.

# Fix: validate reference audio intonation and pitch mean before style extraction
# Require: declarative-only reference, pitch mean within ±3 Hz of speaker neutral mean

import torch
import torchaudio
import numpy as np
import crepe  # F0 extraction
from hierspeech import HierSpeechPP
from hierspeech.style_encoder import StyleEncoder

model = HierSpeechPP.from_pretrained("sh-lee-github/hierspeech-pp-large")
style_encoder = StyleEncoder.from_pretrained("sh-lee-github/hierspeech-pp-large")

PITCH_NEUTRAL_TOLERANCE_HZ = 3.0  # Maximum deviation from speaker neutral mean
MIN_REFERENCE_DURATION_S = 10.0   # Minimum reference duration for stable pitch estimate
RISING_INTONATION_THRESHOLD = 15  # Hz rise in final 20% of utterance = question

def measure_reference_pitch_mean(wav_numpy, sample_rate=22050):
    """
    Measure the mean fundamental frequency of reference audio.
    Returns: (pitch_mean_hz, has_rising_intonation_flag)
    """
    # Extract F0 with CREPE
    time, frequency, confidence, activation = crepe.predict(
        wav_numpy, sample_rate, viterbi=True, model_capacity='medium'
    )

    # Filter: only voiced frames (confidence > 0.5, frequency > 60 Hz)
    voiced_mask = (confidence > 0.5) & (frequency > 60)
    voiced_f0 = frequency[voiced_mask]

    if len(voiced_f0) < 20:
        raise ValueError("Reference audio too short or too unvoiced for pitch analysis")

    pitch_mean = float(np.mean(voiced_f0))

    # Check for rising intonation: does final 20% of voiced F0 rise significantly?
    final_20pct = voiced_f0[int(len(voiced_f0) * 0.8):]
    first_80pct = voiced_f0[:int(len(voiced_f0) * 0.8)]
    final_mean = np.mean(final_20pct)
    body_mean = np.mean(first_80pct)
    has_rising_intonation = (final_mean - body_mean) > RISING_INTONATION_THRESHOLD

    return pitch_mean, has_rising_intonation

def validate_reference_audio(reference_path, speaker_neutral_mean_hz):
    """
    Validate reference audio for HierSpeech++ style extraction.

    Returns the reference if valid; raises ValueError with guidance if not.

    speaker_neutral_mean_hz: pre-measured neutral pitch mean for this speaker
                             (measured from a known declarative speech sample)
    """
    wav, sr = torchaudio.load(reference_path)
    if sr != 22050:
        wav = torchaudio.functional.resample(wav, sr, 22050)

    wav_numpy = wav.squeeze().numpy()
    duration_s = len(wav_numpy) / 22050

    # Check 1: duration
    if duration_s < MIN_REFERENCE_DURATION_S:
        raise ValueError(
            f"Reference too short ({duration_s:.1f}s < {MIN_REFERENCE_DURATION_S}s). "
            f"Provide at least {MIN_REFERENCE_DURATION_S}s of declarative speech."
        )

    # Check 2: rising intonation detection
    pitch_mean, has_rising = measure_reference_pitch_mean(wav_numpy, 22050)

    if has_rising:
        raise ValueError(
            f"Reference contains rising-intonation sentences (detected final F0 rise "
            f"> {RISING_INTONATION_THRESHOLD} Hz). Remove questions from reference clip. "
            f"Use only declarative statements."
        )

    # Check 3: pitch mean proximity to speaker neutral
    offset = pitch_mean - speaker_neutral_mean_hz
    if abs(offset) > PITCH_NEUTRAL_TOLERANCE_HZ:
        raise ValueError(
            f"Reference pitch mean {pitch_mean:.1f} Hz deviates {offset:+.1f} Hz from "
            f"speaker neutral mean {speaker_neutral_mean_hz:.1f} Hz "
            f"(tolerance: ±{PITCH_NEUTRAL_TOLERANCE_HZ} Hz). "
            f"Select a more neutral reference clip."
        )

    print(f"  Reference valid: {duration_s:.1f}s, pitch mean {pitch_mean:.1f} Hz "
          f"(offset {offset:+.1f} Hz from neutral), no rising intonation detected")
    return wav

def extract_style_vector_fixed(reference_path, speaker_neutral_mean_hz):
    """
    Extract validated style vector with pitch-neutral reference audio.
    Prevents style encoder from encoding rising-intonation pitch bias.
    """
    wav = validate_reference_audio(reference_path, speaker_neutral_mean_hz)

    with torch.no_grad():
        style_vector = style_encoder.encode(wav)

    return style_vector  # Clean: pitch mean within ±3 Hz of speaker neutral

def convert_voice_fixed(source_path, reference_path, speaker_neutral_mean_hz):
    """
    Convert source speech to target speaker voice with validated reference.
    Converted output pitch mean matches speaker neutral mean.
    """
    source_wav, sr = torchaudio.load(source_path)
    if sr != 22050:
        source_wav = torchaudio.functional.resample(source_wav, sr, 22050)

    style_vector = extract_style_vector_fixed(reference_path, speaker_neutral_mean_hz)

    with torch.no_grad():
        converted = model.convert(
            source=source_wav,
            style=style_vector,  # Pitch-neutral style vector
        )

    return converted

# ---- Results after declarative reference validation ----
#
# Target speaker pitch offset (50-utterance test set, 10 speakers):
#
#   Speaker 1: neutral mean 118 Hz
#     Before fix: converted mean 133 Hz — offset +15 Hz
#     After fix:  converted mean 119 Hz — offset  +1 Hz ← within tolerance
#
#   Speaker 2: neutral mean 142 Hz
#     Before fix: converted mean 160 Hz — offset +18 Hz
#     After fix:  converted mean 143 Hz — offset  +1 Hz ← within tolerance
#
#   Speaker 3: neutral mean 97 Hz
#     Before fix: converted mean 114 Hz — offset +17 Hz
#     After fix:  converted mean  98 Hz — offset  +1 Hz ← within tolerance
#
# MOS evaluation (20-listener crowdsource, 50 utterances per condition):
#   Before fix: 3.8 (σ=0.4) — good quality, pitch offset perceived as unnatural
#   After fix:  4.0 (σ=0.3) — high quality, pitch matches target speaker
#
# Speaker cosine similarity:
#   Before fix: 0.921 (speaker identity preserved even with pitch offset)
#   After fix:  0.934 (pitch correctness improves perceived speaker similarity)
#
# Reference selection guidance adopted by production team:
#   Required: 10+ second reference clips of declarative paragraph reading
#   Rejected: any clip containing questions (detected by final F0 rise > 15 Hz)
#   Validated: reference pitch mean within ±3 Hz of pre-measured speaker neutral mean

print("Fix: declarative reference audio with validated pitch mean")
print("Pitch offset: 15–20 Hz → <2 Hz; MOS: 3.8 → 4.0")
print("Speaker cosine similarity: 0.921 → 0.934")

Related retainer posts

• VALL-E developer on retainer — neural codec language model TTS; 15–25 ms initial silence gap in voice-cloned outputs; reference codec tokens prepend silence onset; language model matches reference silence; fix: RMS energy threshold trim of reference before codec encoding — 3 weeks
• CosyVoice developer on retainer — flow-matching TTS; cross-lingual mode English speaker voice for Mandarin text; fourth tone starting pitch 10–15 Hz lower than expected; English pitch IQR clips Mandarin fourth-tone starting pitch; fix: PSOLA pitch range normalization to 70 Hz IQR on English reference — 3 weeks
• NaturalSpeech 3 developer on retainer — factorized codec diffusion TTS; progressive duration compression on consecutive unstressed syllables; duration predictor interprets long unstressed sequences as compression artifacts; fix: 65 ms minimum clamp + 8 ms offset for unstressed syllables — 3 weeks