DiffSinger developer on retainer: Mandarin tone 2 and tone 4 vowels drift 20–35 cents flat during syllable duration in bilingual Mandarin/English singing model — shared phoneme embedding space lacks dimensionality to encode tone-directional pitch movement separately from phoneme identity — tone-explicit pitch bias post-processing corrects pitch trajectory — 3 weeks — DiffSinger developer on monthly retainer

DiffSinger (MoonInTheRiver/DiffSinger) diffusion-based singing voice synthesis • Python 3.11; PyTorch 2.1.x • AI vocal synthesis service for music producers • model retrained on bilingual Mandarin/English vocal dataset • Mandarin tone 2 (rising) and tone 4 (falling) syllables: pitch begins at correct note onset but drifts 20–35 cents flat during syllable • tone 2 ends at note midpoint instead of note ceiling; tone 4 starts too high and ends correctly • tone 1 (flat) and tone 3 (dipping) synthesize at correct pitch • developer checks DiffSinger pitch predictor, linguistic feature extractor, and Mandarin phoneme embedding configuration for 3 weeks • root cause: bilingual model uses shared phoneme embedding table for English and Mandarin; English phonemes require no pitch-direction encoding (English not tonal); shared embedding dimensionality insufficient to separate tone-directional features from phoneme identity; pitch predictor conditioned on under-specified tone embeddings undershoots note endpoint for tone 2 and overshoots note onset for tone 4 • fix: tone-explicit pitch bias in post-processing — for tone 2 add +20 to +35 cents to F0 trajectory in final 40% of syllable duration; for tone 4 subtract 20–35 cents in initial 40%; bias magnitude tuned via grid search on 50-syllable test set • pitch drift: 20–35 cents → <5 cents • MOS: 3.5 → 4.0 • 1.5h retainer engagement

Eastfield DiffSinger Ltd. built an AI vocal synthesis service for professional music producers using DiffSinger as the core synthesis engine. The service allowed producers to generate high-quality singing vocals for bilingual Mandarin/English songs: lyrics were submitted with per-syllable note targets and the service returned synthesized vocals that matched the melodic contour. For English lyrics and Mandarin tone 1 and tone 3 syllables, the synthesis was correct — vocals matched the target notes and the pitch trajectories were natural and musically usable. But for Mandarin tone 2 (rising tone, as in má) and tone 4 (falling tone, as in mà) syllables, the synthesized vocals exhibited a systematic pitch artifact: the note onset was correct, but the pitch drifted flat during the syllable, ending 20–35 cents (0.2–0.35 semitones) below the target pitch ceiling. For a rising-tone syllable that should ascend from G3 to B3, the synthesized vocal reached only A♯3 (35 cents flat of B3), producing a note that sounded consistently slightly under-pitched. Music producers described it as “the Mandarin parts sound slightly flat, like the singer never quite reaches the note” and “tone 4 syllables sound like they start too high and then fall correctly, but tone 2 always falls short.”

The team initially checked the DiffSinger pitch predictor configuration: they verified that the note event inputs were correctly formatted (onset pitch, offset pitch, duration all present and correctly quantized) and that the pitch predictor output was being applied before the diffusion process. They also verified the espeak-ng Mandarin phoneme extraction was generating the correct tone markers in the phoneme sequence (e.g., /m a2/ for tone 2 “má” vs /m a4/ for tone 4 “mà”). The tone markers were present. They checked the diffusion model’s conditioning on the pitch predictor output, the vocoder configuration, and the mel spectrogram normalization. None of these explained the systematic tone 2/4 undershoot. After three weeks they engaged a DiffSinger retainer developer who identified the root cause as a phoneme embedding dimensionality limitation: the shared phoneme embedding space for English and Mandarin had insufficient dimensions allocated to tone-directional features, causing the pitch predictor to systematically underestimate the magnitude of pitch movement required for rising and falling tones.

The DiffSinger bilingual model architecture and phoneme embedding analysis at the time of the engagement:

# Eastfield DiffSinger Ltd. — DiffSinger bilingual Mandarin/English singing synthesis
# DiffSinger (MoonInTheRiver/DiffSinger) diffusion-based singing voice synthesis
# Python 3.11, PyTorch 2.1.x

import torch
import numpy as np
from diffsinger import DiffSinger, DiffSingerConfig
from diffsinger.text import phonemize_bilingual, PHONEME_TO_ID

# DiffSinger bilingual configuration:
# Architecture: diffusion-based mel spectrogram generation
# Pitch predictor: Gaussian upsampling + pitch predictor network
# Phoneme embedding: shared table, dim=256, covering both English and Mandarin phonemes
# Language IDs: 0 = English, 1 = Mandarin
# Mandarin tone categories: encoded as suffix markers in phoneme ID space
#   tone 1 (flat):    /a1/, /e1/, /i1/, ... → IDs 300–349
#   tone 2 (rising):  /a2/, /e2/, /i2/, ... → IDs 350–399
#   tone 3 (dipping): /a3/, /e3/, /i3/, ... → IDs 400–449
#   tone 4 (falling): /a4/, /e4/, /i4/, ... → IDs 450–499
# Vocoder: HiFi-GAN (24 kHz)

model = DiffSinger.from_pretrained("eastfield-tts/diffsinger-bilingual-v1")

def synthesize_phrase(phoneme_seq: list, note_onsets: list, note_offsets: list,
                      note_durations: list, language_ids: list):
    """
    Synthesize singing phrase with DiffSinger bilingual model.
    phoneme_seq:    list of phoneme strings, e.g. ['m', 'a2', 'd', 'ao4']
    note_onsets:    per-phoneme note onset pitch (MIDI, float)
    note_offsets:   per-phoneme note offset pitch (MIDI, float)
    note_durations: per-phoneme duration in seconds
    language_ids:   0 = English, 1 = Mandarin

    Bug: for Mandarin tone 2 and tone 4 phonemes, synthesized F0 drifts
    20–35 cents flat relative to target note during syllable.
    """
    input_ids  = torch.tensor([PHONEME_TO_ID[p] for p in phoneme_seq]).unsqueeze(0)
    lang_ids   = torch.tensor(language_ids).unsqueeze(0)
    note_midi  = torch.tensor(note_offsets, dtype=torch.float32).unsqueeze(0)
    note_durs  = torch.tensor(note_durations, dtype=torch.float32).unsqueeze(0)

    with torch.inference_mode():
        # Pitch predictor: generates per-frame F0 conditioned on phoneme embeddings
        # and target note events
        # Bug: for tone 2 (IDs 350–399) and tone 4 (IDs 450–499),
        # phoneme embedding dimensionality insufficient to encode pitch direction
        # → pitch predictor undershoots note target by 20–35 cents
        f0_predicted = model.pitch_predictor(input_ids, lang_ids, note_midi, note_durs)

        # Diffusion: generate mel spectrogram conditioned on predicted F0
        mel = model.diffusion(input_ids, f0_predicted, note_durs)

        # HiFi-GAN: mel → waveform
        waveform = model.vocoder(mel)

    return waveform, f0_predicted

# Diagnosis: pitch predictor output on tone 2 and tone 4 syllables
# Measurement: F0 at syllable endpoint vs target note (MIDI cents from A4=69)
#
# Test phrase: "风 màn màn chuī" (Mandarin tone 4 syllables: màn, màn)
# Target note for each syllable: G4 (67.0 MIDI) → E4 (64.0 MIDI) falling
# (tone 4 falling: starts G4, ends E4)
#
# Pitch predictor output for tone 4 syllable "màn":
#   Frame  Target F0 (Hz)  Predicted F0 (Hz)  Deviation (cents)
#   0      392.0 Hz (G4)   405.6 Hz (+60c)    +60 cents (overshoot onset)
#   10     385.5 Hz        386.2 Hz            +3 cents
#   20     365.5 Hz        367.0 Hz            +7 cents
#   30     329.6 Hz (E4)   347.2 Hz           +91 cents → ~350 Hz actual
#
#   Wait — rechecking: tone 4 falls from onset. Actual bug pattern:
#   Frame 0 (onset):  predicted 405.6 Hz vs target 392.0 Hz (+60c overshot onset)
#   Frame 30 (end):   predicted 345.0 Hz vs target 329.6 Hz (+75c overshot end)
#   Net drift: onset overshot +60c, end overshot +75c — entire trajectory shifted up
#
# Test phrase: "māo zài chàng gē" (tone 2 syllable: ge2 "gé")
# Target note: C4 (60.0 MIDI) → E4 (64.0 MIDI) rising (tone 2 rising)
#
# Pitch predictor output for tone 2 syllable "gé":
#   Frame  Target F0 (Hz)  Predicted F0 (Hz)  Deviation (cents)
#   0      261.6 Hz (C4)   260.1 Hz            -10 cents (onset correct)
#   10     278.0 Hz        270.5 Hz            -47 cents
#   20     303.0 Hz        291.4 Hz            -67 cents
#   30     329.6 Hz (E4)   312.3 Hz            -98 cents ← 98 cents flat at end
#
# Pattern: tone 2 (rising) systematically undershoots top of note by 20–35 cents avg
#          tone 4 (falling) systematically overshoots — entire trajectory shifted up
#
# Tone 1 (flat) and tone 3 (dipping): pitch deviation < 5 cents — correct
#
# Conclusion: tone-directional features poorly represented in shared phoneme embedding space
# English phonemes (IDs 0–299) occupy most of the embedding space and have no
# pitch-direction features; tone 2/4 Mandarin phonemes (IDs 350–499) share the
# space but their directional features are diluted by the English embedding distribution

print("Root cause: shared phoneme embedding dim=256 insufficient for tone-directional encoding")
print("Tone 2 endpoint undershoot: 98 cents at frame 30; tone 4 trajectory shifted up ~60c")
print("Tone 1 and tone 3 unaffected: flat and V-shape captured even with imprecise direction")

Why DiffSinger shared phoneme embeddings fail to encode tone-directional pitch movement for Mandarin tones 2 and 4

DiffSinger’s pitch predictor is a neural network conditioned on phoneme embeddings, with a separate embedding for each phoneme ID in the vocabulary. In a monolingual Mandarin DiffSinger, the embedding dimensionality is sufficient to encode both phoneme identity and tone category, because the entire embedding space is dedicated to Mandarin phonetics including tonal features. In a bilingual English/Mandarin model, the same embedding dimensionality must cover both English phonemes (which require no pitch-direction encoding) and Mandarin phonemes (which require pitch-direction encoding for all four tones). The gradient descent training process distributes the embedding dimensions across all phonemes according to which features are most useful for reducing loss on the training data. Because the training data is mixed English and Mandarin, the embedding dimensions that would encode Mandarin tone-directional features are partially occupied by English phonetic features that happen to co-vary in the training data. The result is that the Mandarin tone 2 and tone 4 embeddings do not fully capture the direction of pitch movement (rising vs. falling), only the magnitude of pitch change.

Tone 1 (flat pitch) is unaffected because a flat pitch trajectory requires no directional encoding — the pitch predictor simply maintains a constant pitch, which requires only the target note value and not the direction of movement. Tone 3 (dipping pitch: falling then rising) is also less affected because the dipping shape is encoded by the temporal structure of the phoneme duration (long enough to include a full fall-and-rise) rather than purely by the direction feature. Tone 2 (monotonically rising) and tone 4 (monotonically falling) are most affected because they require the pitch predictor to consistently move in one direction from onset to offset, which requires a clear directional encoding in the phoneme embedding. The fix is to inject the missing directional information as a post-processing bias rather than retraining the model: after the pitch predictor generates its F0 trajectory, apply a correction that pushes tone 2 trajectories up at the end and tone 4 trajectories down at the start. The bias magnitude (20–35 cents) is determined by grid search over a 50-syllable test set that covers all four tones and multiple note intervals (semitone, third, fifth).

# Fix: tone-explicit pitch bias in DiffSinger pitch predictor post-processing
# Corrects tone 2 (rising) and tone 4 (falling) pitch undershoot in bilingual model

import torch
import numpy as np
from diffsinger import DiffSinger
from diffsinger.text import phonemize_bilingual, PHONEME_TO_ID

model = DiffSinger.from_pretrained("eastfield-tts/diffsinger-bilingual-v1")

# Mandarin tone ID ranges (from DiffSinger bilingual phoneme vocabulary)
TONE_2_IDS = set(range(350, 400))   # rising tone: /a2/, /e2/, /i2/, /o2/, /u2/, ...
TONE_4_IDS = set(range(450, 500))   # falling tone: /a4/, /e4/, /i4/, /o4/, /u4/, ...

# Pitch bias parameters (tuned via grid search on 50-syllable tone 2/4 test set)
# Grid search range: 10–50 cents in 5-cent steps; minimized mean |predicted - target| cents
TONE_2_PITCH_BIAS_CENTS =  28.0   # add 28 cents to F0 in final 40% of tone 2 syllable
TONE_4_PITCH_BIAS_CENTS = -25.0   # subtract 25 cents from F0 in initial 40% of tone 4

def cents_to_f0_ratio(cents: float) -> float:
    """Convert pitch bias in cents to multiplicative F0 ratio."""
    return 2.0 ** (cents / 1200.0)

def apply_tone_pitch_bias(f0_trajectory: torch.Tensor,
                          phoneme_ids: list,
                          frame_to_phoneme: list) -> torch.Tensor:
    """
    Apply tone-explicit pitch bias to DiffSinger F0 trajectory.

    f0_trajectory:    tensor [1, T] of F0 values in Hz (per frame, 24 kHz / hop 128 = 187.5 Hz)
    phoneme_ids:      list of phoneme IDs for each phoneme in the input
    frame_to_phoneme: list mapping frame index → phoneme index

    Returns: corrected F0 trajectory tensor [1, T]
    """
    f0_corrected = f0_trajectory.clone()
    frames_per_phoneme: dict[int, list[int]] = {}

    for frame_idx, phoneme_idx in enumerate(frame_to_phoneme):
        frames_per_phoneme.setdefault(phoneme_idx, []).append(frame_idx)

    for phoneme_idx, pid in enumerate(phoneme_ids):
        if pid not in TONE_2_IDS and pid not in TONE_4_IDS:
            continue   # tone 1 and tone 3: no correction

        frames = frames_per_phoneme.get(phoneme_idx, [])
        if not frames:
            continue

        n_frames = len(frames)
        split_40 = max(1, int(n_frames * 0.40))

        if pid in TONE_2_IDS:
            # Apply upward bias to final 40% of tone 2 syllable (rising: undershoots top)
            bias_ratio = cents_to_f0_ratio(TONE_2_PITCH_BIAS_CENTS)
            for frame_idx in frames[-split_40:]:
                if f0_corrected[0, frame_idx] > 0:   # only voiced frames
                    f0_corrected[0, frame_idx] *= bias_ratio

        elif pid in TONE_4_IDS:
            # Apply downward bias to initial 40% of tone 4 syllable (falling: overshoots onset)
            bias_ratio = cents_to_f0_ratio(TONE_4_PITCH_BIAS_CENTS)
            for frame_idx in frames[:split_40]:
                if f0_corrected[0, frame_idx] > 0:
                    f0_corrected[0, frame_idx] *= bias_ratio

    return f0_corrected

def synthesize_fixed(phoneme_seq: list, note_onsets: list, note_offsets: list,
                     note_durations: list, language_ids: list):
    """Synthesize with DiffSinger + tone-explicit pitch bias correction."""
    input_ids  = torch.tensor([PHONEME_TO_ID[p] for p in phoneme_seq]).unsqueeze(0)
    lang_ids   = torch.tensor(language_ids).unsqueeze(0)
    note_midi  = torch.tensor(note_offsets, dtype=torch.float32).unsqueeze(0)
    note_durs  = torch.tensor(note_durations, dtype=torch.float32).unsqueeze(0)

    with torch.inference_mode():
        f0_raw           = model.pitch_predictor(input_ids, lang_ids, note_midi, note_durs)
        frame_to_phoneme = model.get_frame_to_phoneme_map(input_ids, note_durs)

        # Apply tone-explicit pitch bias (fix)
        f0_fixed = apply_tone_pitch_bias(f0_raw, [PHONEME_TO_ID[p] for p in phoneme_seq],
                                         frame_to_phoneme)

        mel      = model.diffusion(input_ids, f0_fixed, note_durs)
        waveform = model.vocoder(mel)

    return waveform

# ---- Results after tone-explicit pitch bias ----
#
# Pitch deviation from target note (50-syllable test set, 12–13 per tone):
#
#   Tone 2 (rising) — mean pitch deviation at syllable endpoint:
#     Buggy:  −28.4 cents (σ=9.2) — consistently undershoots note ceiling
#     Fixed:   −2.1 cents (σ=3.5) — within perceptual threshold (±5 cents)
#
#   Tone 4 (falling) — mean pitch deviation at syllable onset:
#     Buggy:  +24.7 cents (σ=8.8) — consistently overshoots note onset
#     Fixed:   +1.9 cents (σ=2.9) — within perceptual threshold
#
#   Tone 1 (flat) — mean pitch deviation:
#     Buggy:   +1.3 cents (σ=2.1) — no artifact
#     Fixed:   +1.4 cents (σ=2.1) — no regression (bias not applied to tone 1)
#
#   Tone 3 (dipping) — mean pitch deviation at dip:
#     Buggy:   +3.8 cents (σ=4.0) — slight artifact, within perceptual threshold
#     Fixed:   +3.9 cents (σ=4.1) — no regression (bias not applied to tone 3)
#
# MOS evaluation (20-listener crowdsource, 50 phrases per condition):
#   Buggy bilingual model (Mandarin lyrics): MOS 3.5 (σ=0.5)
#     Listeners: "slightly flat on some syllables", "some notes don't quite reach the top"
#   Fixed bilingual model (Mandarin lyrics): MOS 4.0 (σ=0.3)
#     Listeners: "natural and in-tune", "good pitch accuracy on rising/falling tones"
#
#   English lyrics: MOS 3.9 → 3.9 (no regression — bias not applied to English phonemes)
#
# Grid search results (best bias values on 50-syllable validation set):
#   Tone 2 bias: 28 cents applied to final 40% of syllable duration (range tested: 10–50 cents)
#   Tone 4 bias: 25 cents applied to initial 40% of syllable duration (range tested: 10–50 cents)

print("Fix: tone-explicit pitch bias (tone 2: +28c final 40%; tone 4: -25c initial 40%)")
print("Pitch drift: 20–35 cents → <5 cents; MOS: 3.5 → 4.0 on Mandarin lyrics; 1.5h")

Related retainer posts

Track retainer hours without a client portal

HourTab turns your time-tracker CSV into a public URL your client bookmarks. No login. No portal. CSV in, URL out.

See pricing →