Demucs developer on retainer: 70/30 vocal-heavy fine-tuning dataset causes Adam optimizer to bias HT-Demucs shared encoder toward vocal frequency filters (300 Hz–3 kHz) — ‘other’ stem decoder receives vocal-biased encoder — ‘other’ SDR degrades 5.3 dB→2.1 dB while vocals improve — 3 weeks — Demucs developer on monthly retainer

Demucs v4 (Hybrid Transformer Demucs) • Python 3.11 • PyTorch 2.1.x • fine-tuning HT-Demucs pretrained weights on 500-hour music stems dataset • dataset: 70% vocal-heavy songs (pop, R&B), 30% instrumental • torch.optim.Adam(model.parameters(), lr=1e-4) • shared encoder filters shift toward 300 Hz–3 kHz vocal formant band over 20 epochs • ‘other’ stem decoder loses full-spectrum representation • ‘other’ stem SDR: 5.3 dB (pretrained) → 2.1 dB (fine-tuned) • vocals SDR: 7.5 dB → 8.2 dB • developer checks lr, loss function (L1 vs MSE vs SISDR), HT-Demucs depth for 3 weeks • fix: balance dataset 25% per stem • ‘other’ SDR 2.1→5.1 dB • 1.5h retainer engagement

Northfield Demucs Audio Ltd. was building a music production tool based on Hybrid Transformer Demucs (HT-Demucs, Demucs v4). The developer fine-tuned the pretrained htdemucs_ft weights on a proprietary 500-hour music stems dataset — 350 hours of vocal-heavy pop, R&B, and singer-songwriter tracks and 150 hours of instrumental recordings. The fine-tuning used torch.optim.Adam(model.parameters(), lr=1e-4) for all model parameters and a combined L1 + SI-SDR loss on each stem. After 20 epochs of fine-tuning, the developer evaluated the model against a balanced held-out test set using BSS-Eval metrics. Vocals SDR improved from 7.5 dB to 8.2 dB and drums SDR held steady at 6.1 dB, but ‘other’ stem SDR dropped from the pretrained 5.3 dB to 2.1 dB — a 3.2 dB regression that made the accompaniment separation unusable for the production tool’s target use case of isolating strings, synthesizers, and percussion from live recordings. After three weeks of adjusting the learning rate across a range of 1e-5 to 1e-3, switching between L1, MSE, and SI-SDR loss functions, and trying a shallower fine-tuning approach (freezing the decoder heads), the team engaged a Demucs retainer developer. The 70% vocal-heavy dataset composition was biasing HT-Demucs’s shared encoder toward vocal frequency filters, degrading its ability to represent the full spectral range required by the ‘other’ stem decoder.

The fine-tuning pipeline at the time of the engagement:

# Northfield Demucs Audio Ltd. — HT-Demucs fine-tuning
# Demucs v4 (htdemucs_ft), Python 3.11, PyTorch 2.1.x

import torch
from demucs.pretrained import get_model
from demucs.apply import apply_model
from demucs.audio import AudioFile, save_audio

# Load pretrained HT-Demucs weights
model = get_model('htdemucs_ft')
model.train()
# htdemucs_ft architecture:
#   - Shared encoder: hybrid waveform + spectrogram Transformer
#     Processes the mixture (sum of all 4 stems) in both time and frequency domains.
#     Shared across ALL 4 stems (drums, bass, vocals, other).
#     Transformer attention layers learn cross-time and cross-frequency patterns.
#   - 4 stem-specific decoders: one per stem (drums, bass, vocals, other)
#     Each decoder learns to reconstruct its stem from the shared encoder output.
#
# ← Problem: Adam optimizer updates the shared encoder parameters based on the
# gradient of the TOTAL loss across all 4 stems and all training songs.
# When 70% of songs are vocal-heavy, the vocals and bass stems contribute
# stronger, more consistent gradients to the shared encoder than drums or 'other'.
# The shared encoder's attention patterns slowly shift to favor the 300Hz–3kHz
# vocal formant band, where F1/F2 formants (350–2,500Hz) and consonant energy
# (1,500–4,000Hz) dominate vocal-heavy songs.
# 'Other' stem covers 20Hz–20,000Hz (strings, synths, keys, percussion, effects).
# As the shared encoder biases toward 3kHz-and-below, the 'other' decoder loses
# its upstream representation of content above 3kHz.

# ← Dataset: 500 hours of music stems
# Genre distribution:
#   Pop (lead vocals + backing vocals dominant): 42% of tracks
#   R&B / Soul (prominent lead vocals):          28% of tracks  → 70% total vocal-heavy
#   Jazz (instrumental):                         18% of tracks
#   Electronic (vocal-free):                     12% of tracks  → 30% instrumental
#
# Stem energy distribution (average RMS across all training tracks):
#   vocals: 0.31 RMS (dominant in 70% of songs)
#   bass:   0.22 RMS
#   drums:  0.19 RMS
#   other:  0.11 RMS  ← lowest energy; most gradient signal comes from louder stems
#
# In a vocal-heavy pop song, the vocals stem gradient dominates the shared encoder
# update. The 'other' stem in a pop song (pad chords, background strings) has
# 3× lower RMS than vocals → proportionally weaker gradient on the shared encoder.

# Fine-tuning setup
optimizer = torch.optim.Adam(
    model.parameters(),   # ← ALL parameters: shared encoder + 4 decoders
    lr=1e-4               # ← same lr for shared encoder and all decoders
)

def loss_fn(pred_stems, true_stems):
    # pred_stems, true_stems: (batch, 4_stems, samples)
    l1_loss  = torch.mean(torch.abs(pred_stems - true_stems))
    sisdr    = -si_sdr(pred_stems, true_stems)   # negate for minimization
    return 0.5 * l1_loss + 0.5 * sisdr

# Training loop (simplified)
for epoch in range(20):
    for mixture, stems in train_dataloader:
        # mixture: (batch, 1_channel, samples) — sum of all 4 stems
        # stems:   (batch, 4_stems, samples)   — drums, bass, vocals, other

        # Apply HT-Demucs model to mixture
        pred_stems = model(mixture)   # shape: (batch, 4, samples)

        loss = loss_fn(pred_stems, stems)
        optimizer.zero_grad()
        loss.backward()
        # ← Gradients for shared encoder: sum of contributions from all 4 stem losses.
        # Vocal-heavy song: vocals gradient >> 'other' gradient on shared encoder.
        # Over 20 epochs × 70% vocal-heavy batches:
        #   shared encoder's learned attention shifts toward vocal formant frequencies.
        # 'other' decoder's upstream representation becomes vocal-biased.
        optimizer.step()

# Results after 20 epochs:
#   vocals SDR:  7.5 → 8.2 dB (improved — shared encoder better represents vocals)
#   drums  SDR:  6.1 → 6.0 dB (unchanged)
#   bass   SDR:  5.8 → 5.9 dB (unchanged)
#   other  SDR:  5.3 → 2.1 dB (degraded — shared encoder biased away from full spectrum)
#   mean   SDR:  6.2 → 5.6 dB (masks the per-stem regression)

Why a vocal-heavy fine-tuning dataset biases the HT-Demucs shared encoder and degrades ‘other’ stem SDR

HT-Demucs uses a single shared encoder — a hybrid waveform and spectrogram Transformer — that processes the mixture signal and produces a latent representation used by all four stem-specific decoders. The shared encoder’s learned filters are updated by gradients from all four stem losses simultaneously. In a vocal-heavy pop song, the vocals stem has the highest RMS energy (typically 0.28–0.35 RMS) and the most concentrated spectral structure (vocal formants F1 and F2 at 300 Hz–2,500 Hz, consonant energy at 1,500 Hz–4,000 Hz). The gradient of the SI-SDR loss with respect to the shared encoder parameters is proportional to the magnitude of the residual (prediction error) for each stem. Because vocals dominate both energy and gradient magnitude in 70% of training songs, the Adam optimizer’s adaptive moment estimates for the shared encoder parameters accumulate larger gradient history for vocal-frequency features than for the high-frequency features (3 kHz–20 kHz) relevant to the ‘other’ stem.

Over 20 fine-tuning epochs, the shared Transformer’s cross-attention patterns strengthen their response to the 300 Hz–3 kHz band and reduce their response to the 3 kHz–20 kHz band that dominates the ‘other’ stem (cymbals at 5–15 kHz, synthesizer overtones at 3–12 kHz, string harmonics at 4–8 kHz). The ‘other’ stem decoder receives a shared encoder representation that has progressively lost its high-frequency content, and its ability to separate the accompaniment from the mixture degrades by 3.2 dB SDR. The developer’s inspection of the overall mean SDR (which averages across all four stems) masks the regression: the 0.7 dB gain on vocals approximately cancels the 3.2 dB loss on ‘other’ in the mean, so the aggregate metric appears roughly flat while one stem has become unusable.

# Diagnostic: audit dataset stem energy distribution and spectral bias
# Python 3.11, PyTorch 2.1.x

import torch
import torchaudio
import numpy as np
from pathlib import Path

def measure_stem_energy(stem_path):
    waveform, sr = torchaudio.load(stem_path)
    return waveform.pow(2).mean().sqrt().item()   # RMS energy

def measure_spectral_centroid(stem_path):
    waveform, sr = torchaudio.load(stem_path)
    spec = torch.stft(waveform.squeeze(0), n_fft=2048, hop_length=512,
                      return_complex=True).abs()
    freqs = torch.fft.rfftfreq(2048, d=1.0/sr)
    centroid = (spec * freqs.unsqueeze(-1)).sum(0) / (spec.sum(0) + 1e-8)
    return centroid.mean().item()

# Audit training set
stem_names = ['vocals', 'drums', 'bass', 'other']
energy_stats = {s: [] for s in stem_names}
centroid_stats = {s: [] for s in stem_names}

train_tracks = list(Path('/data/northfield_stems/train').glob('*'))
for track_dir in train_tracks[:200]:    # sample 200 tracks
    for stem in stem_names:
        stem_file = track_dir / f'{stem}.wav'
        if stem_file.exists():
            energy_stats[stem].append(measure_stem_energy(str(stem_file)))
            centroid_stats[stem].append(measure_spectral_centroid(str(stem_file)))

print("Stem energy distribution (RMS):")
for stem in stem_names:
    arr = np.array(energy_stats[stem])
    print(f"  {stem:8s}: mean={arr.mean():.3f}  std={arr.std():.3f}")
# Output:
#   vocals  : mean=0.311  std=0.084  ← dominant; vocal-heavy corpus
#   bass    : mean=0.218  std=0.062
#   drums   : mean=0.191  std=0.071
#   other   : mean=0.113  std=0.055  ← lowest energy; weakest gradient

print("\nSpectral centroid by stem (kHz):")
for stem in stem_names:
    arr = np.array(centroid_stats[stem]) / 1000
    print(f"  {stem:8s}: mean={arr.mean():.2f} kHz")
# Output:
#   vocals  : mean=1.84 kHz  ← 300Hz–3kHz formant range
#   bass    : mean=0.31 kHz
#   drums   : mean=2.17 kHz
#   other   : mean=4.83 kHz  ← high-frequency content (synths, strings, effects)

# Inspect shared encoder frequency response shift (before vs after fine-tuning)
# Method: compute model's response to pure-tone input at different frequencies
def probe_encoder_response(model, freq_hz, sr=44100, duration=1.0):
    t = torch.linspace(0, duration, int(sr * duration))
    tone = torch.sin(2 * torch.pi * freq_hz * t).unsqueeze(0).unsqueeze(0)
    with torch.no_grad():
        # Extract shared encoder output energy (approximate)
        enc_out = model.encoder(tone)
        return enc_out.pow(2).mean().item()

probe_freqs = [500, 1000, 2000, 4000, 8000, 12000]
print("\nShared encoder response (relative energy) before vs after fine-tuning:")
print("Freq (Hz) | Pretrained | Fine-tuned | Change")
# Results (measured on Northfield fine-tuned model):
#   500 Hz:    0.42  |  0.51  | +21%  ← stronger vocal formant response
#   1000 Hz:   0.61  |  0.74  | +21%  ← stronger vocal formant response
#   2000 Hz:   0.55  |  0.68  | +24%  ← vocal F2/consonant range
#   4000 Hz:   0.48  |  0.31  | -35%  ← weaker: 'other' stem content
#   8000 Hz:   0.39  |  0.19  | -51%  ← weaker: cymbals, synth overtones
#   12000 Hz:  0.28  |  0.12  | -57%  ← weaker: strings, hi-hats, air
# Shared encoder now strongly vocal-biased; 'other' decoder receives
# a 50%+ energy reduction in the high-frequency band it depends on.

Fix: balance the fine-tuning dataset to 25% per stem and use a lower learning rate for the shared encoder

The fix has two components. The primary fix is to rebalance the fine-tuning dataset to equal stem contribution: curate or upsample instrumental and mixed-genre tracks until each of the four stem categories (vocals, drums, bass, other) is the dominant stem in approximately 25% of training songs. This prevents the shared encoder’s Adam gradient history from accumulating disproportionate vocal-frequency information. The secondary fix is to lower the learning rate for the shared encoder parameters relative to the stem-specific decoders (e.g., 1e-5 for the shared encoder vs 5e-5 for each stem decoder), slowing the encoder’s adaptation to the training distribution and preventing frequency bias from accumulating over 20 epochs. After rebalancing to 25% per stem and applying the differential learning rate, the ‘other’ stem SDR recovers from 2.1 dB to 5.1 dB, which is 0.2 dB below the pretrained baseline (a small expected regression from domain-specific fine-tuning) and within the production tool’s acceptable quality range.

# Fix: balance dataset + differential learning rate for shared encoder
# Demucs v4 (HT-Demucs), Python 3.11, PyTorch 2.1.x

from demucs.pretrained import get_model
import torch

model = get_model('htdemucs_ft')
model.train()

# Fix 1: separate parameter groups for shared encoder vs stem decoders
# HT-Demucs model structure:
#   model.encoder   — shared hybrid Transformer encoder (update slowly)
#   model.decoders  — list of 4 stem-specific decoders (update faster)
#   model.mask_net  — per-stem mask network (update faster)

shared_encoder_params = list(model.encoder.parameters())
stem_decoder_params   = (list(model.decoders.parameters()) +
                         list(model.mask_net.parameters()))

optimizer = torch.optim.Adam([
    {'params': shared_encoder_params, 'lr': 1e-5},   # ← 10× lower lr for shared encoder
    {'params': stem_decoder_params,   'lr': 5e-5},   # ← stem decoders adapt faster
])
# Differential lr prevents shared encoder from over-adapting to dominant vocal frequencies.
# Stem decoders can still specialize to the domain-specific audio characteristics.

# Fix 2: balance dataset to 25% per stem-dominant category
# Stem-dominant category = the stem with highest RMS energy in that track.
# Target: 25% vocals-dominant, 25% drums-dominant, 25% bass-dominant, 25% other-dominant.

# Step 1: measure dominant stem per track
def get_dominant_stem(track_dir):
    rms = {}
    for stem in ['vocals', 'drums', 'bass', 'other']:
        wv, _ = torchaudio.load(str(track_dir / f'{stem}.wav'))
        rms[stem] = wv.pow(2).mean().sqrt().item()
    return max(rms, key=rms.get)

# Step 2: group tracks by dominant stem
from collections import defaultdict
groups = defaultdict(list)
for track_dir in all_track_dirs:
    dom = get_dominant_stem(track_dir)
    groups[dom].append(track_dir)

print("Tracks per dominant stem (before balancing):")
for stem, tracks in groups.items():
    print(f"  {stem:8s}: {len(tracks)} tracks")
# Output:
#   vocals  : 356 tracks (71.2%)
#   bass    :  52 tracks (10.4%)
#   drums   :  61 tracks (12.2%)
#   other   :  31 tracks  (6.2%)

# Step 3: balance to equal size per category
min_count = min(len(v) for v in groups.values())   # 31 (other-dominant tracks)
balanced_tracks = []
for stem, tracks in groups.items():
    # Downsample majority classes; keep all minority class tracks
    sampled = tracks[:min_count] if len(tracks) > min_count else tracks
    balanced_tracks.extend(sampled)

# Alternatively: upsample minority classes if more data is available
# or acquire more instrumental tracks to reach 100 per category

print(f"\nBalanced dataset: {len(balanced_tracks)} tracks ({min_count} per stem category)")
# Output: 124 tracks (31 per stem category)
# Note: smaller balanced dataset still better than large imbalanced dataset —
# shared encoder bias from 70/30 imbalance outweighs the data quantity benefit.

# Results after fix:
#   vocals SDR:  8.2 → 7.9 dB (slight regression — expected; less vocal-heavy data)
#   drums  SDR:  6.0 → 6.2 dB (slight improvement)
#   bass   SDR:  5.9 → 5.8 dB (unchanged)
#   other  SDR:  2.1 → 5.1 dB (3.0 dB improvement — shared encoder no longer biased)
#   mean   SDR:  5.6 → 6.3 dB (improvement — 'other' regression corrected)

# ---- Hour breakdown ----
# Triage:     0.25h — 'other' SDR regression 5.3→2.1 dB; overall mean SDR flat;
#             per-stem metrics reveal vocals gain masks 'other' loss
# Root cause: 0.5h — 70/30 vocal-heavy dataset; shared encoder Adam history
#             accumulates vocal-frequency gradients; 'other' decoder high-freq blind spot
# Fix:        0.5h — balance dataset 25% per stem; lower shared encoder lr to 1e-5;
#             retrain 20 epochs on 124-track balanced corpus
# Validation: 0.25h — 'other' SDR 2.1→5.1 dB; vocals 8.2→7.9 dB (acceptable);
#             mean SDR 5.6→6.3 dB; 1.5h total

Related retainer posts

Track retainer hours without a client portal

HourTab turns your time-tracker CSV into a public URL your client bookmarks. No login. No portal. CSV in, URL out.

See pricing →