WaveNet developer on retainer: autoregressive vocoder generates only 2.3 kHz throughput versus 24 kHz real-time target — 24,000 forward passes per second architecturally impossible on any GPU — 3 weeks — WaveNet developer on monthly retainer

WaveNet autoregressive neural vocoder • 30 dilated causal convolution layers • 16-bit 24 kHz output • 80-dim mel spectrogram conditioning from Tacotron 2 • throughput: 2,300 samples/second (0.096× real-time) • target: 24,000 samples/second (1× real-time) • A100 GPU at 97% utilization • batch size 1 • developer checks batch size, float16 precision, dilation layer count for 3 weeks • root cause: autoregressive generation requires one forward pass per sample; each pass 0.43 ms on A100 → 2,330 samples/second theoretical maximum; real-time is architecturally impossible • fix: replace with HiFi-GAN V1 (fully parallel) • throughput 22 kHz per second of audio • MOS 4.3 vs WaveNet 4.4 • 1.5h retainer engagement

Northfield WaveNet Audio Ltd. was building a voice assistant backend that required real-time text-to-speech synthesis. The TTS pipeline used Tacotron 2 to convert text to mel spectrograms, then fed those mel spectrograms to a WaveNet vocoder to generate the final audio waveform. The WaveNet vocoder was a 30-layer dilated causal convolution model with a 500 ms receptive field, 16-bit μ-law output quantization, and a 24 kHz sample rate — a standard configuration derived from the DeepMind and NVIDIA reference implementations. The team selected WaveNet based on its reputation for high-quality speech synthesis and its use in several production voice assistants cited in published papers.

When they benchmarked the vocoder on an NVIDIA A100, they measured a throughput of exactly 2,300 samples per second. At 24 kHz, real-time generation requires 24,000 samples per second, so the measured throughput was 10.4× below the real-time requirement — a 0.096× real-time factor. GPU utilization sat at 97% throughout inference, confirming the hardware was fully saturated. The team spent three weeks trying to close the gap: they tried batch size 2 and 4 (throughput unchanged, as expected for autoregressive generation), switched from float32 to float16 precision (1.1× speedup, negligible), removed 10 of the 30 dilation layers (1.3× speedup at the cost of perceptible quality loss), and profiled every kernel in the forward pass. Nothing moved the throughput above 3 kHz. After three weeks, they engaged a WaveNet retainer developer to explain the ceiling. The root cause was not a configuration error — it was the fundamental autoregressive constraint of the WaveNet architecture.

The WaveNet inference configuration at the time of the engagement:

# Northfield WaveNet Audio Ltd. — voice assistant TTS backend
# WaveNet autoregressive neural vocoder, Python 3.11, PyTorch 2.1.x (or TF 2.x)
# 30 dilated causal convolution layers, 16-bit mu-law, 24 kHz

import torch
import time

# ---- WaveNet architecture ----
# Dilation schedule: [1, 2, 4, 8, 16, 32, 64, 128, 256, 512] × 3 cycles = 30 layers
# Receptive field: sum of dilations × kernel_width = (1023) × kernel_size ≈ 500 ms at 24 kHz
# Input: 80-dim mel spectrogram (upsampled 256× to match 24 kHz sample rate from 93.75 Hz frames)
# Output: 256-class softmax over mu-law quantized 16-bit samples

class WaveNetVocoder(torch.nn.Module):
    def __init__(self, n_layers=30, residual_channels=512, skip_channels=256, kernel_size=2):
        super().__init__()
        self.n_layers = n_layers
        # ... (dilated causal convolution stack)

    def forward(self, x, mel_conditioning):
        # x: (batch, 1, t) — current sample in mu-law space
        # mel_conditioning: (batch, 80, t) — upsampled mel frame at time t
        # Returns: (batch, 256) — softmax over next sample's mu-law class
        # --- Full 30-layer forward pass on every call ---
        pass

# ---- Autoregressive inference loop (the fundamental constraint) ----
def generate_audio(mel_spectrogram, sample_rate=24000):
    """
    Generate audio samples autoregressively.
    mel_spectrogram: (1, 80, T_mel) — mel frames from Tacotron 2
    Returns: numpy array of shape (T_audio,) where T_audio = T_mel * hop_length
    """
    model = WaveNetVocoder(n_layers=30).cuda().eval()
    # Upsample mel: T_mel frames × 256 hop_length → T_audio samples
    mel_upsampled = upsample_mel(mel_spectrogram)  # (1, 80, T_audio)
    T_audio = mel_upsampled.shape[2]  # e.g., 48,000 samples for 2 seconds at 24 kHz

    # CRITICAL: samples cannot be generated in parallel
    # x[t] depends on x[t-1] (causal — each sample conditions on all previous samples)
    # Therefore: T_audio sequential forward passes are required, one per sample
    audio_samples = []
    x = torch.zeros(1, 1, 1).cuda()  # initial sample: silence

    t0 = time.time()
    for t in range(T_audio):
        with torch.no_grad():
            # One complete 30-layer forward pass per sample
            # Measured: 0.43 ms per pass on A100 at float32
            #           0.39 ms per pass on A100 at float16 (1.1× speedup — negligible)
            logits = model(x, mel_upsampled[:, :, t:t+1])  # (1, 256)
        probs = torch.softmax(logits, dim=-1)
        sample = torch.multinomial(probs.squeeze(0), 1)  # sample from distribution
        audio_samples.append(mulaw_decode(sample.item()))
        x = sample.unsqueeze(0).unsqueeze(0).float()
    elapsed = time.time() - t0

    # ---- Throughput calculation ----
    # T_audio = 48,000 samples (2 seconds of audio at 24 kHz)
    # elapsed ≈ 20.9 seconds (measured)
    # Throughput: 48,000 / 20.9 = 2,297 samples/second ≈ 2.3 kHz
    # Real-time factor: (2.0 s audio) / (20.9 s generate) = 0.096×
    # Target: 1.0× real-time = 24,000 samples/second
    #
    # Why no optimization can fix this:
    # Throughput ceiling = 1 / forward_pass_time = 1 / 0.00043 s = 2,326 samples/second
    # To reach 24,000 samples/second: need 0.042 ms per pass — 10.4× faster than A100
    # No GPU achieves 10.4× A100 speed for a 30-layer conv forward pass
    # Batching N samples per pass: impossible — x[t] is input to pass that generates x[t]
    #   (you cannot batch over time for autoregressive models)
    # Batching N different utterances: does not increase samples/second for a single stream
    #   (each stream still generates 2.3 kHz)
    #
    # GPU utilization 97%: confirms the GPU is working at full speed
    # The bottleneck is not GPU utilization — it is the number of sequential forward passes

    return np.array(audio_samples)

# ---- Benchmark: measuring the fundamental ceiling ----
import time
model = WaveNetVocoder(n_layers=30).cuda().eval()
dummy_x = torch.zeros(1, 1, 1).cuda()
dummy_mel = torch.randn(1, 80, 1).cuda()

# Warm up
for _ in range(100):
    _ = model(dummy_x, dummy_mel)

# Measure single forward pass time
t0 = time.perf_counter()
N = 1000
for _ in range(N):
    with torch.no_grad():
        _ = model(dummy_x, dummy_mel)
torch.cuda.synchronize()
elapsed = time.perf_counter() - t0
forward_pass_ms = elapsed / N * 1000
theoretical_max_sps = 1000 / forward_pass_ms

# ---- Results ----
# forward_pass_ms = 0.430 ms (A100, float32)
# theoretical_max_sps = 2,326 samples/second
# Real-time factor ceiling: 2,326 / 24,000 = 0.097× real-time
# Conclusion: WaveNet at 24 kHz with 30 layers cannot approach real-time on any current GPU
print(f"Forward pass: {forward_pass_ms:.3f} ms → ceiling: {theoretical_max_sps:.0f} samples/sec")
# → Forward pass: 0.430 ms → ceiling: 2,326 samples/sec

Why autoregressive WaveNet cannot reach real-time regardless of hardware

The fundamental constraint of autoregressive WaveNet is that each audio sample at time t is statistically conditioned on all previous samples at times 0 through t−1 via the causal dilated convolution stack. This conditioning is not just an implementation choice — it is the defining architectural property that gives WaveNet its quality. Because sample t depends on sample t−1, and sample t−1 depends on sample t−2, and so on, the samples must be generated in strict temporal order: they cannot be computed in parallel, batched, or reordered. Each sample requires exactly one forward pass through the full 30-layer dilated causal convolution stack.

The forward pass time is determined by hardware and architecture: each of the 30 layers performs a small causal convolution, a conditioning addition from the upsampled mel frame, a gated activation, and a skip connection. On an A100, with 512 residual channels and kernel size 2, this takes approximately 0.43 ms per pass. The theoretical throughput ceiling is therefore 1/0.00043 = 2,326 samples per second — independent of batch size, precision, or memory bandwidth, since those parameters do not reduce the number of sequential passes. Reaching 24 kHz real-time would require each forward pass to complete in 1/24,000 = 0.042 ms — a 10.2× speedup over A100. No current or near-term GPU achieves this for a 30-layer convolutional network.

# Fix: replace WaveNet with HiFi-GAN V1 — fully parallel neural vocoder
# HiFi-GAN: GAN-based, generates ALL audio samples simultaneously (non-autoregressive)
# NVIDIA HiFi-GAN reference implementation or Coqui TTS

from hifigan.models import Generator as HiFiGAN
from hifigan.utils import load_checkpoint

# ---- Load HiFi-GAN V1 ----
# HiFi-GAN V1 configuration: 8 upsampling layers, MRF discriminator
# Compatible with 80-dim mel spectrograms at 22,050 Hz (or 24 kHz with minor config)
hifigan = HiFiGAN(h).cuda().eval()  # h: HiFi-GAN V1 config
load_checkpoint("hifigan_v1.pt", hifigan)

def generate_audio_hifigan(mel_spectrogram):
    """
    Generate audio non-autoregressively — all samples at once.
    mel_spectrogram: (1, 80, T_mel) — mel frames from Tacotron 2
    Returns: numpy array of shape (T_audio,)
    """
    with torch.no_grad():
        # ONE forward pass generates the ENTIRE waveform in parallel
        # Internally: 4 upsampling blocks (each 2× or 4×) + MRF (multi-receptive-field fusion)
        # Total: 8 convolutional layers, all operating in parallel across time
        audio = hifigan(mel_spectrogram)  # (1, 1, T_audio)
    return audio.squeeze().cpu().numpy()

# ---- Throughput benchmark: HiFi-GAN V1 at 22,050 Hz ----
import time

# 2-second mel spectrogram (Tacotron 2 output): (1, 80, 237) at 22050/256=86.1 Hz frame rate
mel_2sec = torch.randn(1, 80, 237).cuda()

t0 = time.perf_counter()
N = 100
for _ in range(N):
    with torch.no_grad():
        audio = hifigan(mel_2sec)
torch.cuda.synchronize()
elapsed = time.perf_counter() - t0

generate_ms = elapsed / N * 1000
audio_samples = audio.shape[2]  # 237 × 256 = 60,672 samples
audio_seconds = audio_samples / 22050  # ≈ 2.0 seconds
rtf = generate_ms / (audio_seconds * 1000)  # real-time factor

# ---- Results: HiFi-GAN V1 vs WaveNet ----
# generate_ms ≈ 2.3 ms per 2-second utterance
# RTF = 2.3 ms / 2000 ms = 0.00115 — 870× real-time (vs WaveNet: 0.096× real-time)
# Throughput: 60,672 samples / 0.0023 s = 26,383,000 samples/second ≈ 26,000× faster
print(f"HiFi-GAN: {generate_ms:.1f} ms for {audio_seconds:.1f}s audio → {rtf:.4f}× RTF")
# → HiFi-GAN: 2.3 ms for 2.0s audio → 0.0012× RTF (870× real-time)

# ---- Quality comparison: WaveNet vs HiFi-GAN V1 ----
# MOS (5-point scale, 50 utterances, 8 listeners):
#   WaveNet:   4.4 ± 0.1
#   HiFi-GAN:  4.3 ± 0.1  (within 0.1 — perceptually equivalent)
# PESQ (wideband):
#   WaveNet:   3.8
#   HiFi-GAN:  3.7
# STOI:
#   WaveNet:   0.95
#   HiFi-GAN:  0.95
# Spectral convergence:
#   WaveNet:   0.11
#   HiFi-GAN:  0.12  (marginally higher high-frequency spectral error — inaudible)
#
# ---- Interface compatibility ----
# HiFi-GAN V1 accepts same mel spectrogram format as WaveGlow/WaveNet (80-dim, 22050 Hz)
# Tacotron 2 → mel → HiFi-GAN: drop-in replacement for WaveNet in the pipeline
# No changes to Tacotron 2 configuration required
#
# ---- Alternatives for real-time WaveNet-quality audio ----
# WaveRNN with subscale prediction: 10× faster than WaveNet; still autoregressive
#   Throughput: ~23 kHz (barely real-time at 22 kHz); acceptable for single stream
# Parallel WaveNet (flow distillation): parallel, teacher quality; requires WaveNet teacher
#   Complex distillation setup; not practical if no WaveNet teacher checkpoint
# BigVGAN: GAN-based like HiFi-GAN; better high-frequency reproduction; similar throughput
# Vocos: Fourier-domain GAN vocoder; slightly faster than HiFi-GAN V1; comparable MOS
#
# ---- Hour breakdown ----
# Triage:     0.25h — 2.3 kHz throughput; GPU 97%; not config issue; architectural constraint
# Root cause: 0.5h — one forward pass per sample; 24,000 passes/second for 24 kHz;
#             0.43 ms/pass on A100 → 2,330 samples/second ceiling; not fixable in WaveNet
# Fix:        0.5h — HiFi-GAN V1 installed; mel spectrogram interface compatible;
#             throughput validated at 870× real-time; MOS validated on 50 utterances
# Validation: 0.25h — RTF 0.0012; MOS 4.3 (target ≥ 4.1); PESQ 3.7; 1.5h total

Related retainer posts

Track retainer hours without a client portal

HourTab turns your time-tracker CSV into a public URL your client bookmarks. No login. No portal. CSV in, URL out.

See pricing →