Open-Unmix developer on retainer: n_fft=4096 STFT window (92.9 ms at 44.1 kHz) insufficient to represent 143–200 ms vibrato cycles in Broadway and opera vocals — UMX masking model treats vibrato-rich harmonics as accompaniment noise — SDR 3.2 dB vs expected 7+ dB on musical test set — 3 weeks — Open-Unmix developer on monthly retainer

Open-Unmix (umxhq) 1.2.x • Python 3.11 • PyTorch 2.1.x • fine-tuning on 400-hour Broadway and opera musical dataset • openunmix.umxhq(targets=['vocals']) • n_fft=4096, hop_length=1024 (defaults) • window duration 4096 / 44100 = 92.9 ms • vibrato 5–7 Hz (cycle 143–200 ms) • window captures <65% of vibrato cycle • harmonics smear 10–30 Hz per bin during vibrato • UMX masks vibrato passages as noise • SDR on classical musical test set: 3.2 dB (expected 7+ dB for vocals-only) • developer checks n_fft 2048 / 4096 / 8192, UMX architecture for 3 weeks • fix: n_fft=8192 (186 ms window captures full vibrato cycle) • SDR 3.2→7.4 dB • 1.5h retainer engagement

Northfield OpenUnmix Sep Ltd. was fine-tuning Open-Unmix (the umxhq pretrained model) for a custom vocal stem separation task on a 400-hour dataset of musical theater recordings — Broadway cast albums and opera recordings sourced from commercially licensed stems. The developer loaded the umxhq model using openunmix.umxhq(targets=['vocals']) and fine-tuned only the vocals-target network on the custom musical dataset using the default STFT configuration: n_fft=4096 and hop_length=1024, inherited from the pretrained model. Training proceeded for 100 epochs on a single A100 GPU. After fine-tuning, evaluation on a standard pop/rock test set (MUSDB18-HQ) produced SDR of 7.2 dB, confirming that the fine-tuning had not degraded pop vocal separation. However, evaluation on a held-out musical theater and opera test set produced SDR of only 3.2 dB against an expected 7+ dB for vocals-only separation with UMX. The low SDR was concentrated in sustained notes with active vibrato, opera high notes, and ornamental melismatic passages. After three weeks of experimenting with different n_fft values (2048, 4096, 8192), adjusting training data augmentation, and reviewing the UMX architecture, the team engaged an Open-Unmix retainer developer. The root cause was that the default STFT window of 92.9 ms was insufficient to represent the 143–200 ms vibrato cycles present throughout the musical theater and opera vocals, causing UMX to mask vibrato-rich vocal passages as accompaniment noise.

The Open-Unmix configuration at the time of the engagement:

# Northfield OpenUnmix Sep Ltd. — musical theater vocal separation
# Open-Unmix (umxhq) 1.2.x, Python 3.11, PyTorch 2.1.x

import openunmix
import torch
import torchaudio

# ---- Model loading ----
# Load pretrained umxhq with default STFT parameters
model = openunmix.umxhq(targets=['vocals'])
# umxhq default STFT configuration:
#   n_fft=4096, hop_length=1024, sample_rate=44100
#   window_length = n_fft = 4096 samples
#   window_duration = 4096 / 44100 Hz = 92.9 ms
#   frequency_resolution = 44100 / 4096 ≈ 10.8 Hz per bin

# ---- Musical corpus: vocal ornament timescales ----
# Broadway / opera vocals (400-hour training set):
#
#   Vibrato:
#     Frequency: 5–7 Hz (typical Western classical / musical theater vibrato)
#     Cycle duration: 1/5 Hz = 200 ms (slow) to 1/7 Hz = 143 ms (fast)
#     Pitch oscillation depth: ±50–150 cents around the nominal pitch
#
#   Fundamental frequency of musical theater vocals:
#     Soprano range: 250–1200 Hz
#     Tenor range: 130–520 Hz
#     During vibrato at 5 Hz on a soprano G5 (784 Hz):
#       Pitch oscillates ±50 cents → ±28 Hz around 784 Hz
#       Harmonic n oscillates ±28n Hz:
#         Harmonic 1: 784 ± 28 Hz (784–756 to 784+28 = 756–812 Hz)
#         Harmonic 2: 1568 ± 56 Hz
#         Harmonic 5: 3920 ± 140 Hz
#
#   STFT window analysis (THE PROBLEM):
#     window_duration = 92.9 ms
#     vibrato_cycle_duration = 143–200 ms (5–7 Hz vibrato)
#     window captures 92.9 / 200 = 46.5% to 92.9 / 143 = 65% of a vibrato cycle.
#
#     Within a 92.9 ms window, the singer's fundamental oscillates through
#     46–65% of its vibrato trajectory without completing the cycle.
#     The DFT of an incomplete sinusoidal frequency sweep leaks energy
#     into neighboring frequency bins:
#       Leakage bandwidth ≈ ±f_vibrato × n_harmonic bins
#       For soprano G5 harmonic 5 during 5 Hz vibrato:
#         Leakage ≈ ±(28 Hz × 5) / (10.8 Hz/bin) ≈ ±13 bins
#         → harmonic 5 smears across ~26 bins (280 Hz bandwidth)
#
#     UMX's spectrogram masking network was trained on pop vocals
#     with narrow, stable harmonics. It learns:
#       stable_narrow_band → vocal (high mask)
#       smeared_broad_band → accompaniment (low mask)
#     ← Vibrato-rich harmonics are broad-band → masked as accompaniment.
#
# ---- Evaluation results ----
# SDR on MUSDB18-HQ pop test set:  7.2 dB  (fine-tuning didn't hurt pop)
# SDR on musical theater test set: 3.2 dB  (expected 7+ dB)
#   Breakdown by passage type:
#     Vibrato-free (spoken, consonants): 7.1 dB  ← UMX works fine here
#     Vibrato-heavy (sustained vowels, high notes): 0.8 dB  ← near baseline
#
# Developer checks n_fft 2048/4096/8192, training augmentation, UMX architecture for 3 weeks.
# Never connects low SDR specifically to vibrato-heavy passages.

Why STFT window length must exceed the vibrato cycle duration

Open-Unmix operates on the Short-Time Fourier Transform (STFT) magnitude spectrogram: the input audio is divided into overlapping windows, each of length n_fft samples, and the DFT is computed within each window. The frequency resolution of each DFT bin is sample_rate / n_fft Hz. The temporal resolution of the spectrogram is determined by the hop length (hop_length / sample_rate seconds per frame). The STFT is implicitly a trade-off: longer windows produce finer frequency resolution but coarser temporal resolution; shorter windows produce finer temporal resolution but coarser frequency resolution.

The critical requirement for vocal source separation with spectral masking is that the STFT window must be long enough to represent the periodic structure of the vocal signal as a coherent pattern of narrow spectral peaks. For pop and rock vocals (without significant vibrato), this is typically satisfied by n_fft=4096 at 44.1 kHz (92.9 ms window): sustained vowel sounds maintain near-constant pitch within a 92.9 ms window, producing sharp spectral peaks that UMX learns to identify as vocal signal. For musical theater and opera vocals with vibrato, the requirement is different: the vocal fundamental oscillates at 5–7 Hz (cycle 143–200 ms), meaning the pitch completes one full oscillation cycle every 143–200 ms. A 92.9 ms STFT window captures only 46–65% of a vibrato cycle. Within such a partial window, the vocal fundamental is a frequency-swept tone, not a steady sinusoid. The DFT of a frequency-swept tone produces spectral leakage: energy spreads across neighboring frequency bins proportional to the sweep range divided by the frequency resolution. For a soprano high note at 784 Hz with ±50 cent vibrato depth and 10.8 Hz/bin frequency resolution, the fundamental smears across approximately ±3 bins, and the fifth harmonic smears across approximately ±13 bins, creating a 280 Hz-wide energy smear instead of a narrow spectral peak.

# Diagnostic: compare STFT spectrogram for vibrato vs non-vibrato passage
# Open-Unmix 1.2.x, Python 3.11, PyTorch 2.1.x

import torch
import torchaudio
import matplotlib
matplotlib.use('Agg')
import matplotlib.pyplot as plt
import numpy as np

def compute_stft_magnitude(audio, n_fft, hop_length, sample_rate=44100):
    # Compute STFT magnitude spectrogram
    window = torch.hann_window(n_fft)
    stft = torch.stft(
        audio.squeeze(0),
        n_fft=n_fft,
        hop_length=hop_length,
        win_length=n_fft,
        window=window,
        return_complex=True,
    )
    return stft.abs()

# Load a 4-second soprano passage from the test set
audio, sr = torchaudio.load("test/soprano_vibrato_G5.wav")
# audio: soprano sustained G5 (784 Hz) with 5 Hz vibrato, ±50 cents

# Compare n_fft=4096 (default umxhq) vs n_fft=8192 (fix)
mag_4096 = compute_stft_magnitude(audio, n_fft=4096, hop_length=1024)
mag_8192 = compute_stft_magnitude(audio, n_fft=8192, hop_length=1024)

# Measure spectral spread around the fundamental at 784 Hz
# Bin index for 784 Hz:
bin_4096 = int(784 / (44100 / 4096))   # ≈ 72
bin_8192 = int(784 / (44100 / 8192))   # ≈ 145

# Energy in ±10 bins around fundamental (vibrato passage):
spread_4096 = mag_4096[bin_4096-10:bin_4096+10, :].mean().item()
peak_4096 = mag_4096[bin_4096, :].mean().item()

spread_8192 = mag_8192[bin_8192-10:bin_8192+10, :].mean().item()
peak_8192 = mag_8192[bin_8192, :].mean().item()

print(f"n_fft=4096: peak/spread ratio = {peak_4096/spread_4096:.2f}")
print(f"n_fft=8192: peak/spread ratio = {peak_8192/spread_8192:.2f}")
# Output (Northfield soprano G5 vibrato passage):
#   n_fft=4096: peak/spread ratio = 1.21  ← harmonics smeared, not peaked
#   n_fft=8192: peak/spread ratio = 3.84  ← harmonics sharper (window > cycle)

# A peak/spread ratio near 1.0 means the energy is smeared uniformly across
# the ±10-bin neighborhood — no distinct peak for UMX to identify as vocal.
# A higher ratio means energy is concentrated in the peak bin — UMX can
# distinguish vocal harmonics from accompaniment spectra.

Fix: increase n_fft to cover at least one full vibrato cycle

The fix is to increase n_fft from 4096 to 8192 samples. At 44.1 kHz, n_fft=8192 produces a window duration of 8192 / 44100 ≈ 185.8 ms — sufficient to cover one full vibrato cycle at 5 Hz (200 ms) at 92% completion, and a full cycle at 6 Hz (167 ms) or 7 Hz (143 ms). Within a window that spans at least one vibrato cycle, the DFT of the oscillating vocal fundamental averages over the full pitch sweep, producing a frequency smear whose bandwidth is determined by the vibrato depth (pitch range) rather than the incomplete-cycle leakage. More importantly, the smearing is consistent and predictable: the harmonics appear as characteristic broadened peaks whose width encodes the vibrato depth, not as noise-like spread. After fine-tuning UMX with n_fft=8192 on the same 400-hour musical dataset, SDR on the musical theater and opera test set improved from 3.2 to 7.4 dB. SDR specifically on vibrato-heavy passages improved from 0.8 to 7.1 dB, confirming that the STFT window length was the root cause. SDR on the MUSDB18-HQ pop test set remained at 7.0 dB (down only 0.2 dB from 7.2), since pop vocals with minimal vibrato are equally well-represented by both window lengths.

# Fix: increase n_fft to cover full vibrato cycle
# Open-Unmix 1.2.x, Python 3.11, PyTorch 2.1.x

import openunmix
import torch

# Option A: load umxhq and modify STFT parameters before fine-tuning
model = openunmix.umxhq(targets=['vocals'])

# Replace the STFT transform with n_fft=8192
# In Open-Unmix 1.2.x, the STFT is defined in openunmix/transforms.py
# as make_filterbanks(n_fft, hop_length, center, sample_rate, device)
# Patch the transform on the loaded model:

from openunmix.transforms import make_filterbanks
n_fft_new = 8192       # 185.8 ms window at 44.1 kHz → covers full 5 Hz vibrato cycle
hop_length_new = 1024  # keep same hop length (23.2 ms frame shift)

new_encoder, new_decoder = make_filterbanks(
    n_fft=n_fft_new,
    n_hop=hop_length_new,
    center=True,
    sample_rate=44100,
    device=next(model.parameters()).device,
)

# Patch the model's STFT encoder and iSTFT decoder
model.encoder = new_encoder
model.decoder = new_decoder
# Note: the input dimension of the first linear layer in the UMX network
# depends on n_fft // 2 + 1 (number of STFT bins). Changing n_fft
# requires also adjusting the first layer's input size.
# With n_fft=4096: 2049 bins; with n_fft=8192: 4097 bins.
# Either reload umxhq with n_fft=8192 from scratch (untrained) and
# fine-tune from scratch on the musical dataset, or use the approach below:

# Option B: construct UMX from scratch with n_fft=8192 for fine-tuning
# (recommended for musical corpora — start from scratch with correct window)
from openunmix import model as umx_model

# Build a fresh umxhq architecture with n_fft=8192
model_8192 = umx_model.OpenUnmix(
    input_mean=None,
    input_scale=None,
    nb_channels=2,
    hidden_size=512,
    n_fft=8192,          # ← 185.8 ms window
    n_hop=1024,
    max_bin=None,
    sample_rate=44100,
)
# Fine-tune this model on the 400-hour musical dataset.

# STFT window selection guidelines for musical styles:
#
# Target vibrato frequency → minimum n_fft at 44.1 kHz
#   7 Hz vibrato (143 ms cycle) → n_fft ≥ 44100×0.143 = 6,306 → use n_fft=8192
#   5 Hz vibrato (200 ms cycle) → n_fft ≥ 44100×0.200 = 8,820 → use n_fft=8192
#   3 Hz vibrato (333 ms cycle) → n_fft ≥ 44100×0.333 = 14,685 → use n_fft=16384
#   Pop vocals (no vibrato)      → n_fft=4096 sufficient (92.9 ms default)
#
# Note: increasing n_fft increases model input dimension and GPU memory.
# n_fft=8192 roughly doubles the first-layer weight matrix vs n_fft=4096.

# Results after fine-tuning with n_fft=8192 for 100 epochs:
# SDR on musical theater / opera test set: 7.4 dB  (up from 3.2 dB)
# SDR on MUSDB18-HQ pop test set:          7.0 dB  (down 0.2 dB from 7.2 — negligible)
# SDR on vibrato-heavy passages:           7.1 dB  (up from 0.8 dB)
#
# ---- Hour breakdown ----
# Triage:     0.25h — SDR 3.2 dB; breakdown by passage type;
#             vibrato-free passages 7.1 dB vs vibrato-heavy 0.8 dB
# Root cause: 0.5h — n_fft=4096 → 92.9 ms window; vibrato cycle 143–200 ms;
#             window <1 cycle; harmonics smear → UMX masks as noise
# Fix:        0.5h — n_fft=8192 (185.8 ms window > full vibrato cycle);
#             rebuild and retrain fine-tuning run on musical dataset
# Validation: 0.25h — SDR 3.2→7.4 dB; vibrato SDR 0.8→7.1 dB; 1.5h total

Related retainer posts

Track retainer hours without a client portal

HourTab turns your time-tracker CSV into a public URL your client bookmarks. No login. No portal. CSV in, URL out.

See pricing →