AudioCraft / MusicGen developer on retainer: generate_with_chroma chroma conditioning exhausted after 150 frames (3 s reference) — melody conditioning fails after frame 150 of 1,500 — 3 weeks — MusicGen developer on monthly retainer

audiocraft 1.3.0 • Python 3.11 • PyTorch 2.1.x • MusicGen.get_pretrained('facebook/musicgen-melody') • model.set_generation_params(duration=30) • generate_with_chroma with 3-second piano reference at 44.1 kHz • 3 s → 150 chroma frames at 50 Hz • generated output follows reference melody only in first 3 s • frames 151–1,500: chroma conditioning exhausted → unconditioned text-to-music generation • chroma cosine similarity: seconds 0–3 = 0.78, seconds 3–30 = 0.21 • developer checks reference melodies, text prompts, duration settings for 3 weeks • fix: melody_wavs = reference.repeat(1, 1, math.ceil(30 / 3)) loops 3-s melody to 30 s • chroma cosine similarity seconds 0–30 = 0.74 • 1.5h retainer engagement

Northfield AudioCraft Music Ltd. was building an AI music generation system using MusicGen for conditional music generation in a video game soundtrack pipeline. The system accepted a text description of the desired music style and a short reference melody recorded by the game’s composer, then generated a 30-second music segment that combined the specified orchestral style with the composer’s melodic motif. The developer loaded the model with model = MusicGen.get_pretrained('facebook/musicgen-melody'), configured generation duration with model.set_generation_params(duration=30), and called model.generate_with_chroma(descriptions=['epic orchestral battle music'], melody_wavs=reference_audio, melody_sample_rate=44100, progress=True). The reference audio was a 3-second piano melody recorded at 44.1 kHz.

The generated output had correct text-conditioned style — the orchestral instrumentation and battle-music energy matched the description well — but the melody conditioning was inconsistent. During the first 3 seconds, the generated music followed the reference melody closely. Starting at second 3, the generated music diverged completely from the reference melody and continued as unconditioned text-to-music generation for the remaining 27 seconds. The developer tried different reference melodies (piano, violin, hummed), different text prompts, and different duration settings (15, 20, 30 seconds) over three weeks. The pattern persisted: the first N seconds matched the reference, where N was the reference audio duration, and all subsequent seconds diverged. After three weeks, the team engaged a MusicGen retainer developer. The root cause was that generate_with_chroma’s chroma conditioning window is bounded by the reference audio duration — a 3-second reference produces only 150 chroma frames at MusicGen’s 50 Hz chroma frame rate, leaving frames 151–1,500 (seconds 3–30) unconditioned.

The MusicGen generation configuration at the time of the engagement:

# Northfield AudioCraft Music Ltd. — melody-conditioned music generation
# audiocraft 1.3.0, Python 3.11, PyTorch 2.1.x

import torch
import torchaudio
from audiocraft.models import MusicGen

# ---- Load MusicGen melody model ----
model = MusicGen.get_pretrained('facebook/musicgen-melody')
# musicgen-melody is the only MusicGen variant that supports generate_with_chroma
# Other variants (musicgen-small, musicgen-medium, musicgen-large) do not support
# melody conditioning and will raise an error if generate_with_chroma is called.

model.set_generation_params(
    duration=30,       # 30 seconds of output
    top_k=250,
    temperature=1.0,
    cfg_coef=3.0,
)

# ---- Load reference melody ----
# 3-second piano melody recorded by the game composer at 44.1 kHz stereo
reference_wav, ref_sr = torchaudio.load("composer_melody.wav")
# reference_wav shape: (2, 132300) — stereo, 44.1 kHz, 3 seconds
# Duration: 132300 / 44100 = 3.0 seconds

print(f"Reference duration: {reference_wav.shape[1] / ref_sr:.1f} seconds")
# Output: Reference duration: 3.0 seconds

# ---- Generate melody-conditioned music ----
# generate_with_chroma internally:
# 1. Resamples reference_wav to 32 kHz (MusicGen's internal sample rate)
# 2. Extracts 12-dimensional ChromaFeatures at 50 Hz (1 chroma frame per 20 ms)
# 3. For a 3-second reference: 3.0 × 50 = 150 chroma frames
# 4. These 150 chroma frames condition the autoregressive token generation
#    at token positions 0–149 (each token = 20 ms)
# 5. At token positions 150–1499 (seconds 3–30): no chroma conditioning available
#    → generation reverts to pure text-conditioned (unconditioned by melody)

wav = model.generate_with_chroma(
    descriptions=['epic orchestral battle music'],
    melody_wavs=reference_wav.unsqueeze(0),  # (1, 2, 132300) — batch dim added
    melody_sample_rate=ref_sr,
    progress=True,
)
# wav shape: (1, 1, 1323000) — 30 seconds at 44.1 kHz

# ---- Measure melody consistency ----
# Chroma cosine similarity between generated output and reference melody,
# measured in 3-second windows:
# Window 0–3 s:   chroma cosine similarity = 0.78 (good — melody is followed)
# Window 3–6 s:   chroma cosine similarity = 0.23 (poor — melody diverges)
# Window 6–9 s:   chroma cosine similarity = 0.19
# Window 9–12 s:  chroma cosine similarity = 0.22
# Window 12–30 s: chroma cosine similarity = 0.20 (unconditioned text-to-music)
#
# Developer checks for 3 weeks:
# 1. Different reference melodies (violin, hummed, synthesizer):
#    melody follows for N seconds (N = reference duration), then diverges
# 2. Different text prompts (ambient, jazz, electronic):
#    same pattern — first N seconds conditioned, rest unconditioned
# 3. duration=15 with 3-second reference:
#    melody follows 3 s, diverges for 12 s
# 4. duration=5 with 3-second reference:
#    melody follows 3 s, diverges for 2 s
# Pattern: conditioning window = reference audio duration, not generation duration.
# Developer does not investigate the chroma frame count vs generation frame count.

Why chroma conditioning runs out after the reference audio duration

MusicGen’s melody conditioning works by extracting a chromagram from the reference audio — a time-frequency representation of pitch content across 12 chroma classes (C, C#, D, D#, E, F, F#, G, G#, A, A#, B) at 50 Hz time resolution — and using those chroma features as soft conditioning signals at each autoregressive generation step. At each token position t, if a chroma frame exists for that position, the model biases its token distribution toward musically consonant tokens. If no chroma frame exists for position t, the model generates that token using only the text conditioning signal.

The number of chroma frames produced by the reference audio is round(reference_duration × 50): a 3-second reference produces 150 frames covering positions 0–149 (seconds 0–3). A 30-second generation at 50 Hz token rate produces 1,500 token positions. For a 3-second reference, positions 0–149 are chroma-conditioned and positions 150–1,499 are unconditioned. The developer’s observation — that the generated music follows the reference melody during the first 3 seconds and diverges completely thereafter — is an exact symptom of this conditioning gap. The fix is to loop the reference audio to match the generation duration before passing it to generate_with_chroma, so that all 1,500 generation positions have a corresponding chroma frame.

# Fix: loop reference audio to match generation duration
# audiocraft 1.3.0, Python 3.11, PyTorch 2.1.x

import math
import torch
import torchaudio
from audiocraft.models import MusicGen

model = MusicGen.get_pretrained('facebook/musicgen-melody')
model.set_generation_params(duration=30, top_k=250, temperature=1.0, cfg_coef=3.0)

reference_wav, ref_sr = torchaudio.load("composer_melody.wav")
# reference_wav shape: (2, 132300) — stereo, 44.1 kHz, 3.0 seconds
ref_duration = reference_wav.shape[1] / ref_sr  # 3.0 seconds
generation_duration = 30                         # seconds

# ---- Fix: repeat reference audio to cover the full generation duration ----
n_repeats = math.ceil(generation_duration / ref_duration)
# 3.0 s reference, 30 s generation → ceil(30 / 3) = 10 repeats
looped_reference = reference_wav.repeat(1, n_repeats)
# looped_reference shape: (2, 1323000) — 30 seconds at 44.1 kHz (10× repeated melody)

print(f"Original reference:  {reference_wav.shape[1] / ref_sr:.1f} s → {round(ref_duration * 50)} chroma frames")
print(f"Looped reference:    {looped_reference.shape[1] / ref_sr:.1f} s → {round(generation_duration * 50)} chroma frames")
# Output:
# Original reference:  3.0 s → 150 chroma frames
# Looped reference:    30.0 s → 1500 chroma frames

wav = model.generate_with_chroma(
    descriptions=['epic orchestral battle music'],
    melody_wavs=looped_reference.unsqueeze(0),  # (1, 2, 1323000)
    melody_sample_rate=ref_sr,
    progress=True,
)
# wav shape: (1, 1, 1323000) — 30 seconds at 44.1 kHz

# ---- Results after fix ----
# Chroma cosine similarity between generated output and reference melody:
# Window 0–3 s:   0.76 (reference melody period 1)
# Window 3–6 s:   0.72 (reference melody period 2 — same melody, looped)
# Window 6–9 s:   0.74 (reference melody period 3)
# Window 9–30 s:  0.71–0.78 (all chroma-conditioned — melody consistently followed)
# Overall (0–30 s): chroma cosine similarity = 0.74 (up from 0.21 for seconds 3–30)
#
# Listening test: the 3-second piano motif is audible throughout all 30 seconds of
# the generated orchestral output; the motif repeats every 3 seconds with orchestral
# instrumentation variations as expected from the text conditioning.
#
# ---- Alternative: use a longer reference ----
# If a looped melody sounds repetitive (some use cases), provide a reference audio
# >= generation duration that evolves naturally (e.g., a 30-second piano improvisation
# on the motif) — the chroma conditioning will follow the full 30-second melodic arc.
#
# ---- When to use generate vs generate_with_chroma ----
# generate(): text conditioning only; use when no melodic reference is available
# generate_with_chroma(): text + chroma conditioning; use when a melodic motif
#   must be reproduced in the generated output; always provide reference audio
#   >= generation duration for consistent full-duration melody conditioning
#
# ---- Hour breakdown ----
# Triage:     0.25h — melody follows reference in first 3 s, diverges from second 3;
#             chroma cosine similarity 0.78 (sec 0–3) vs 0.21 (sec 3–30);
#             consistent with chroma conditioning window exhaustion at frame 150
# Root cause: 0.5h — generate_with_chroma chroma frame count = reference_duration × 50;
#             3-s reference → 150 chroma frames covering positions 0–149;
#             30-s generation → 1,500 positions; positions 150–1499 unconditioned
# Fix:        0.5h — reference.repeat(1, math.ceil(duration / ref_duration));
#             looped reference: 30 s → 1,500 chroma frames; full conditioning coverage
# Validation: 0.25h — chroma cosine similarity 0.21→0.74 (sec 3–30); 1.5h total

Related retainer posts

Track retainer hours without a client portal

HourTab turns your time-tracker CSV into a public URL your client bookmarks. No login. No portal. CSV in, URL out.

See pricing →