Voicebox developer on retainer: CNF with num_steps=10 produces pre-echo artifacts on stop consonants — Euler ODE integration error overshoots sharp amplitude transients — ghost burst 30–50 ms before /p/ /t/ /k/ onset — 3 weeks — Voicebox developer on monthly retainer

Meta Voicebox reference implementation • Python 3.11 • PyTorch 2.1.x • continuous normalizing flow (CNF) conditioned on masked audio • num_steps=10 (10 Euler ODE solver steps) • generated speech: ASR WER 0.08 (correct text) • ECAPA-TDNN similarity 0.69 (correct speaker identity) • pre-echo artifact: ghost consonant burst 30–50 ms before /p/, /t/, /k/ onset • absent on sustained vowels and fricatives • developer checks reference audio samples, text inputs, temperature settings for 3 weeks • fix: num_steps=10 → num_steps=50 • truncation error 25× smaller • pre-echo eliminated • 1.5h retainer engagement

Westfield Voicebox TTS Ltd. was building a zero-shot TTS pipeline using Meta’s Voicebox continuous normalizing flow (CNF) model. The system encoded a reference speaker utterance as masked audio conditioning, aligned the input text at the phoneme level, and ran the CNF inference from a Gaussian noise prior to the speech waveform using an Euler ODE solver with num_steps=10. The choice of 10 steps was deliberate: the team was targeting low-latency TTS for a voice assistant application, and 10 Euler steps was the minimum they found that produced intelligible speech in preliminary testing. The generated speech passed their automated quality checks: ASR WER was 0.08 (correct text content) and ECAPA-TDNN speaker embedding cosine similarity to the reference speaker was 0.69 (above their 0.65 acceptance threshold for speaker identity transfer).

When the team moved from automated metrics to careful perceptual evaluation, they found a consistent artifact on stop consonants. Every utterance containing /p/, /t/, or /k/ had a faint “pre-echo” — a ghost of the consonant burst that appeared approximately 30–50 ms before the actual burst onset. The artifact was most pronounced on /p/ and /t/ stops, slightly less so on /k/, and entirely absent on sustained vowels, fricatives (/s/, /f/), and nasals (/m/, /n/). To listeners, it sounded like a subtle doubling of plosive consonants or a faint reverb that preceded rather than followed the burst. After three weeks checking different reference audio samples, different text inputs, and different temperature parameters, the team engaged a Voicebox retainer developer. The root cause was a numerical artifact of the Euler ODE solver at low step counts operating on a flow trajectory with sharp curvature near stop consonant amplitude transients.

The Voicebox inference configuration at the time of the engagement:

# Westfield Voicebox TTS Ltd. — zero-shot TTS inference pipeline
# Meta Voicebox reference implementation, Python 3.11, PyTorch 2.1.x

import torch
from voicebox import VoiceboxModel, CNFSolver, MaskedConditioner

# ---- Load Voicebox model ----
model = VoiceboxModel.from_pretrained("facebook/voicebox-base")
model.eval()

# ---- Prepare masked audio conditioning ----
# Reference speaker: 5-second utterance at 16 kHz
reference_wav, sr = load_audio("reference_speaker.wav")  # (1, 80000)
# Extract mel spectrogram of reference (80-dim mel, 50 Hz frame rate)
reference_mel = extract_mel(reference_wav, sr=16000)     # (1, T_ref, 80)

# Mask the first 2 seconds of the reference → Voicebox generates content
# for the masked region conditioned on the unmasked context (last 3 seconds)
mask = torch.ones(1, reference_mel.shape[1], dtype=torch.bool)
mask[:, :100] = False  # mask first 100 frames (2 seconds at 50 Hz)
# Voicebox's in-context learning: unmasked frames provide speaker identity conditioning
# masked frames are what the CNF generates

# ---- Text alignment ----
phonemes = text_to_phonemes("Please put the parcel on the table.")
# phoneme sequence: [p, l, iː, z, p, ʊ, t, ð, ə, p, ɑː, s, ə, l, ɒ, n, ð, ə, t, eɪ, b, ə, l]
# Align phoneme durations to mel frame positions (CTC forced alignment)
aligned_phonemes = align_phonemes(phonemes, reference_mel)

# ---- CNF inference: Euler ODE solver with num_steps=10 ----
solver = CNFSolver(method="euler", num_steps=10)
# num_steps=10 → step_size = 1.0 / 10 = 0.1
# ODE integration from t=0 (Gaussian noise) to t=1 (speech mel spectrogram)
# Each Euler step: x_{t+h} = x_t + h * flow_model(x_t, t, conditioning)
# Truncation error per step: O(h^2) = O(0.01) — large for sharp transients

with torch.no_grad():
    generated_mel = solver.integrate(
        model=model,
        conditioning_mel=reference_mel,
        mask=mask,
        phoneme_alignment=aligned_phonemes,
        noise=torch.randn_like(reference_mel),
    )
    # generated_mel: (1, 100, 80) — mel spectrogram for the masked 2-second segment

# ---- Vocode mel to waveform ----
generated_wav = vocode_mel(generated_mel)  # HiFi-GAN vocoder

# ---- Quality evaluation ----
# ASR WER: 0.08 — transcription correct (text content reproduced)
# ECAPA-TDNN cosine similarity (generated vs reference speaker): 0.69 (target ≥ 0.65)
# Perceptual evaluation: pre-echo on /p/, /t/, /k/ — ghost burst 30–50 ms before onset
# /p/ in "Please", "put", "parcel": pre-echo audible on all three
# /t/ in "put", "table": pre-echo audible
# /k/ in (not in this example): lighter artifact vs /p/ and /t/
# Sustained vowels (iː, ʊ, eɪ): no artifact
# Developer checks for 3 weeks:
# 1. Reference audio (different speakers, different recording conditions): same pre-echo
# 2. Text inputs (without plosives): pre-echo absent; with plosives: pre-echo present
# 3. Temperature (0.5, 0.7, 1.0): same pre-echo pattern at all temperatures
# Never investigates num_steps or ODE solver truncation error.

Why Euler ODE truncation error at num_steps=10 causes pre-echo on stop consonants

Voicebox generates speech by integrating a continuous normalizing flow from a Gaussian noise prior (t=0) to the speech mel spectrogram (t=1) using an ODE solver. The Euler method approximates the ODE trajectory with discrete steps: at each step t, the solver evaluates the flow model’s vector field and takes a step of size h = 1/num_steps in the direction of the vector field. The local truncation error per step is proportional to h² × |d²x/dt²| — the step size squared times the second derivative of the trajectory (its curvature). For smooth portions of the speech signal (sustained vowels, fricatives), the trajectory curvature is low and the h=0.1 step size introduces negligible error. For stop consonants, the trajectory has a sharp inflection point near the onset frame: the ODE must transition rapidly from silence (or low-energy context) to a high-energy burst in a small time window. At this inflection point, |d²x/dt²| is large, and the O(h²) = O(0.01) truncation error is large enough to cause the solver to overshoot the true trajectory.

The overshoot manifests as a pre-echo because of how the Euler method handles the trajectory near the onset: at the step immediately before the burst frame, the large step carries the trajectory partially into the burst before the onset time, depositing a small-amplitude echo of the burst energy in the preceding frame. The following step corrects back toward the true trajectory, but the correction arrives 1–2 steps (20–40 ms at the mel frame rate) after the initial overshoot — placing the corrective step at or slightly after the true burst onset. The net result in waveform space is a ghost burst at t−30 to t−50 ms and the main burst at the correct onset time t.

# Fix: increase num_steps=10 → num_steps=50
# Meta Voicebox reference implementation, Python 3.11, PyTorch 2.1.x

# ---- CNF inference: Euler ODE solver with num_steps=50 ----
solver = CNFSolver(method="euler", num_steps=50)
# num_steps=50 → step_size = 1.0 / 50 = 0.02
# Truncation error per step: O(h^2) = O(0.0004)
# Relative to num_steps=10: (0.02/0.1)^2 = 0.04 → 25× smaller truncation error
# Near stop consonant onset (high curvature), error is now below perceptual threshold

with torch.no_grad():
    generated_mel_50 = solver.integrate(
        model=model,
        conditioning_mel=reference_mel,
        mask=mask,
        phoneme_alignment=aligned_phonemes,
        noise=torch.randn_like(reference_mel),
    )

generated_wav_50 = vocode_mel(generated_mel_50)

# ---- Results after num_steps=10 → num_steps=50 ----
# ASR WER:                   0.08 → 0.07 (slight improvement — better onset timing)
# ECAPA-TDNN similarity:     0.69 → 0.71 (slight improvement)
# Pre-echo on /p/, /t/, /k/: eliminated (blind perceptual test, 50 utterances, 8 listeners)
# Latency:                   5× higher (50 Euler steps vs 10)
#   At 16 kHz, 2-second generation: 10 steps ≈ 180 ms; 50 steps ≈ 900 ms
#   Acceptable for non-real-time TTS (audiobook, dubbing, batch TTS)
#   Not acceptable for <200 ms real-time TTS (requires flow distillation)
#
# ---- Why num_steps=50 is sufficient ----
# Stop consonant burst onset ≤ 5 ms (measured on corpus)
# At 50 Hz mel frame rate: burst onset spans ≈ 1 frame
# With num_steps=50, step_size=0.02 → Euler error at onset curvature peak
#   below perceptual threshold (measured at ~0.3 dB SPL equivalent)
# Increasing to num_steps=100 produces no further perceptual improvement
# → num_steps=50 is the minimum for pre-echo-free stop consonant synthesis
#
# ---- Alternative: use midpoint or RK4 solver instead of Euler ----
# CNFSolver(method="midpoint", num_steps=20):
#   Midpoint method: O(h^3) truncation error (3rd order, vs Euler 2nd order)
#   num_steps=20, step_size=0.05 → error O(0.000125) — comparable to Euler num_steps=50
#   20 midpoint steps: 2 NFE (network function evaluations) each = 40 total NFE
#   50 Euler steps: 50 NFE — same latency for equivalent acoustic quality
#   Use midpoint if you need low total NFE with pre-echo suppression
#
# CNFSolver(method="rk4", num_steps=10):
#   RK4: O(h^5) truncation error (5th order)
#   num_steps=10, step_size=0.1 → error O(0.00001) — well below perceptual threshold
#   10 RK4 steps: 4 NFE each = 40 total NFE — same latency as 40 Euler steps
#   Pre-echo eliminated even at num_steps=10 (but 4 NFE/step vs 1 NFE/step for Euler)
#
# ---- Hour breakdown ----
# Triage:     0.25h — pre-echo on /p/ /t/ /k/ only; absent on vowels/fricatives;
#             ASR WER 0.08 and ECAPA-TDNN 0.69 correct; artifact is in solver, not model
# Root cause: 0.5h — Euler ODE truncation error O(h^2); num_steps=10 → h=0.1;
#             stop consonant onset curvature high → overshoot before burst frame;
#             ghost burst 30–50 ms pre-echo; Euler step corrects after onset
# Fix:        0.5h — num_steps=10 → num_steps=50; h=0.02; truncation error 25× smaller;
#             pre-echo eliminated; latency 5×, acceptable for non-real-time TTS
# Validation: 0.25h — blind perceptual test; 50 utterances high plosive density;
#             8 listeners; 0/50 pre-echo detections at num_steps=50; 1.5h total

Related retainer posts

Track retainer hours without a client portal

HourTab turns your time-tracker CSV into a public URL your client bookmarks. No login. No portal. CSV in, URL out.

See pricing →