WaveGrad developer on retainer: long silent segments produce 60–80 Hz tonal hum artifact instead of silence — U-Net conditioning on near-zero mel frames carries residual low-frequency feature statistics bias into reverse diffusion — silence-mask hard-zeroing silent waveform segments eliminates hum — 3 weeks — WaveGrad developer on monthly retainer

WaveGrad (ivanvovk/WaveGrad) diffusion-based neural vocoder • Python 3.11; PyTorch 2.1.x • real-time TTS pipeline • mel spectrograms with long silent segments (>0.5 seconds of silence mid-utterance, as occurs in em-dash pauses or long comma pauses) produce a 60–80 Hz tonal hum artifact during the silent segment instead of silence • hum absent with HiFi-GAN from same mels • developer checks WaveGrad noise schedule, beta sequence, and conditioning for 3 weeks • root cause: WaveGrad U-Net conditioned on mel spectrogram; during reverse diffusion for near-zero mel frames, network’s learned feature statistics carry residual low-frequency bias; bias manifests as 60–80 Hz tonal artifact in reconstructed waveform at silence • fix: silence mask — detect silent mel frames (mean energy < −60 dBFS) and hard-zero corresponding waveform segments • hum rate: 100% of long-silent segments → 0% • MOS: 3.5 → 4.0 • 1.5h retainer engagement

Westfield WaveGrad Ltd. deployed WaveGrad as the neural vocoder in their real-time TTS pipeline, replacing an earlier HiFi-GAN vocoder that had been limited by occasional spectral inconsistencies on fast-speech inputs. WaveGrad’s diffusion-based generation offered better theoretical guarantees on waveform quality and was showing improved MOS scores on the team’s standard evaluation set. But a new artifact appeared that had not been present with HiFi-GAN: whenever the upstream TTS acoustic model generated a mel spectrogram containing a long silent segment — more than half a second of near-zero energy mid-utterance — WaveGrad produced a faint but audible 60–80 Hz tonal hum during the silence instead of near-perfect quiet.

The hum was consistently present in em-dash pauses (“The deadline is Friday — not negotiable”), long comma pauses in complex sentences, and explicit silence tokens in the acoustic model output. It was not random noise — it was a pure tonal artifact that listeners described as a low electrical hum or a faint motor sound. Critically, synthesizing the identical mel spectrograms through the original HiFi-GAN produced clean silence in the same positions. The artifact was introduced entirely by WaveGrad’s reverse diffusion process.

The team initially hypothesized that the beta noise schedule was miscalibrated for the low-energy silent mel frames — that the schedule was not stepping the noise down adequately at the end of diffusion for near-silent inputs. They tried reducing the minimum beta value and increasing the number of diffusion steps for the final 20% of iterations. Neither change eliminated the hum, though both affected MOS on voiced segments. After three weeks of tuning the noise schedule and conditioning parameters without resolving the artifact, they engaged a WaveGrad retainer developer who traced the root cause to residual low-frequency bias in the U-Net’s feature statistics under near-zero mel conditioning.

The WaveGrad pipeline and silence artifact diagnostic at the time of the engagement:

# Westfield WaveGrad Ltd. — WaveGrad 60-80 Hz hum during long silent segments
# WaveGrad (ivanvovk/WaveGrad) diffusion-based neural vocoder
# Python 3.11, PyTorch 2.1.x
# Deployment: real-time TTS pipeline

import torch
import numpy as np
from wavegrad import WaveGrad  # ivanvovk/WaveGrad

SAMPLE_RATE = 22050
HOP_LENGTH = 256  # mel frame hop in samples (11.6 ms per frame)
SILENCE_THRESHOLD_DBFS = -60.0

checkpoint = torch.load('wavegrad_checkpoint.pt')
model = WaveGrad()
model.load_state_dict(checkpoint['model'])
model.eval()

# --- DIAGNOSTIC: measure RMS energy in synthesized silence ---

def dbfs_rms(waveform_segment: np.ndarray) -> float:
    """Compute dBFS RMS energy of a waveform segment."""
    rms = np.sqrt(np.mean(waveform_segment ** 2))
    if rms == 0.0:
        return -np.inf
    return 20.0 * np.log10(rms)


def find_silent_mel_frames(
    mel: np.ndarray,
    threshold_dbfs: float = SILENCE_THRESHOLD_DBFS,
    frame_window: int = 5
) -> list:
    """
    Return (start_frame, end_frame) pairs for silent mel regions.
    A frame is silent if its mean energy across mel bins is < threshold_dbfs.
    Groups consecutive silent frames into regions.
    """
    # mel shape: (n_mels, n_frames)
    frame_energy_dbfs = []
    for i in range(mel.shape[1]):
        mean_lin = np.mean(np.exp(mel[:, i]))  # mel is log-mel
        if mean_lin == 0.0:
            frame_energy_dbfs.append(-np.inf)
        else:
            frame_energy_dbfs.append(20.0 * np.log10(mean_lin))

    silent_frames = [i for i, e in enumerate(frame_energy_dbfs)
                     if e < threshold_dbfs]

    # Group consecutive silent frames into regions
    regions = []
    if not silent_frames:
        return regions
    start = silent_frames[0]
    prev = silent_frames[0]
    for f in silent_frames[1:]:
        if f - prev > frame_window:
            regions.append((start, prev))
            start = f
        prev = f
    regions.append((start, prev))
    return regions


# Test case: mel with 0.6-second silence mid-utterance
# (simulating em-dash pause: "The deadline is Friday — not negotiable")
mel_with_silence = load_mel_from_acoustic_model("The deadline is Friday — not negotiable")

with torch.no_grad():
    waveform_wavegrad = model.forward(
        torch.from_numpy(mel_with_silence).unsqueeze(0).float()
    ).squeeze(0).numpy()

waveform_hifigan = hifigan_model.infer(mel_with_silence)  # control

# Find the silent mel region (0.6s of near-zero frames mid-utterance)
silent_regions = find_silent_mel_frames(mel_with_silence, threshold_dbfs=-60.0)
# Expected: [(silent_start_frame, silent_end_frame)] — approximately 52 frames (0.6s)

for (s_frame, e_frame) in silent_regions:
    s_sample = s_frame * HOP_LENGTH
    e_sample = (e_frame + 1) * HOP_LENGTH

    wavegrad_segment = waveform_wavegrad[s_sample:e_sample]
    hifigan_segment = waveform_hifigan[s_sample:e_sample]

    wavegrad_rms = dbfs_rms(wavegrad_segment)
    hifigan_rms = dbfs_rms(hifigan_segment)

    print(f"Silent region (frames {s_frame}-{e_frame}, "
          f"{(e_frame - s_frame + 1) * HOP_LENGTH / SAMPLE_RATE:.2f}s):")
    print(f"  WaveGrad RMS in silence: {wavegrad_rms:.1f} dBFS")
    print(f"  HiFi-GAN RMS in silence: {hifigan_rms:.1f} dBFS")

# Output:
# Silent region (frames 38-89, 0.60s):
#   WaveGrad RMS in silence: -41.3 dBFS  ← 60-80 Hz hum at -41 dBFS
#   HiFi-GAN RMS in silence: -74.8 dBFS  ← clean silence at -74 dBFS
#
# Spectral analysis of WaveGrad silent segment:
#   FFT peak at 68 Hz at amplitude -38 dBFS  ← tonal hum confirmed
#   No energy above 200 Hz (not broadband noise)
#   Hum present on 100% of long-silent (>0.5s) inputs
#   Hum absent on short silences (<0.1s): RMS -68 to -72 dBFS
#
# Root cause: U-Net residual low-frequency bias in near-zero mel conditioning
# Hum duration scales with silent segment length (more diffusion steps = more bias accumulation)

Why WaveGrad’s U-Net produces residual 60–80 Hz bias during reverse diffusion of near-zero mel frames

WaveGrad’s reverse diffusion process iteratively estimates the score (the gradient of the log probability density with respect to the noisy waveform) using a U-Net backbone. At each diffusion step, the U-Net receives the current noisy waveform and the mel spectrogram conditioning signal for the corresponding time positions, and outputs a gradient estimate that is used to step toward the denoised waveform. The mel spectrogram conditioning is applied at multiple resolution levels of the U-Net through feature-wise linear modulation (FiLM) or similar conditioning mechanisms, ensuring that the generated waveform follows the spectral structure specified by the mel frames.

For voiced speech and fricatives, the mel conditioning signal provides strong, information-rich guidance: the mel frames carry clear spectral structure that the U-Net encodes into its intermediate feature representations, and the resulting gradient estimate reliably points toward the target waveform. For silent segments, however, the mel conditioning signal is near-zero across all mel bins — the acoustic model is specifying that this region should contain no speech. The U-Net’s convolutional layers process this near-zero signal, but their intermediate activations are not zero: learned bias terms in the convolutional layers, batch normalization statistics trained predominantly on voiced frames, and the residual connections in the U-Net architecture all contribute nonzero activations even when the conditioning input is near-silent.

These residual activations introduce a low-frequency bias into the score estimate for the silent frames. The specific frequency of the bias — 60–80 Hz — reflects the learned spectral statistics of the training data: at low frequencies, the network’s feature maps have characteristic activation patterns from training on speech waveforms where low-frequency energy (fundamental frequency, formant resonance, environmental noise) is statistically correlated with certain conditioning patterns. When the conditioning is near-zero (silence), these low-frequency correlations are not suppressed because the near-zero conditioning does not provide a sufficient signal to override the learned statistical bias. Each diffusion step accumulates a small contribution to the low-frequency bias. Over many steps on a 0.6-second silent region, the cumulative bias becomes audible as a tonal hum.

# Fix: silence mask — detect silent mel frames and hard-zero corresponding waveform segments
# Applied after WaveGrad inference, before audio output
# No model modification required

import torch
import numpy as np

SAMPLE_RATE = 22050
HOP_LENGTH = 256
SILENCE_THRESHOLD_DBFS = -60.0
SILENCE_MIN_FRAMES = 5  # only mask regions with >= 5 consecutive silent frames (~58ms)
MASK_FADE_SAMPLES = 64  # 2.9 ms linear fade to avoid click at mask boundary


def build_silence_mask(
    mel: np.ndarray,
    n_samples: int,
    threshold_dbfs: float = SILENCE_THRESHOLD_DBFS,
    min_frames: int = SILENCE_MIN_FRAMES,
    fade_samples: int = MASK_FADE_SAMPLES
) -> np.ndarray:
    """
    Build a waveform-domain silence mask from mel spectrogram energy.
    Returns an array of shape (n_samples,) with values in [0, 1].
    Silent regions are 0, voiced regions are 1, with short linear fade transitions.
    """
    n_frames = mel.shape[1]
    # Compute mean log-mel energy per frame, convert to dBFS
    frame_energy_lin = np.exp(mel).mean(axis=0)  # (n_frames,)
    frame_energy_dbfs = np.where(
        frame_energy_lin > 0,
        20.0 * np.log10(frame_energy_lin),
        -np.inf
    )

    is_silent = frame_energy_dbfs < threshold_dbfs

    # Identify silent regions >= min_frames consecutive frames
    mask_frames = np.ones(n_frames, dtype=float)
    i = 0
    while i < n_frames:
        if is_silent[i]:
            j = i
            while j < n_frames and is_silent[j]:
                j += 1
            if (j - i) >= min_frames:
                mask_frames[i:j] = 0.0
            i = j
        else:
            i += 1

    # Upsample frame mask to sample domain (nearest-neighbor)
    mask_samples = np.repeat(mask_frames, HOP_LENGTH)[:n_samples]

    # Apply linear fades at mask boundaries to prevent clicks
    for i in range(1, len(mask_samples)):
        if mask_samples[i] != mask_samples[i - 1]:
            # Transition detected: apply fade
            if mask_samples[i] == 0.0:
                # Fade in to silence (rising mask edge)
                start = max(0, i - fade_samples)
                fade = np.linspace(1.0, 0.0, i - start)
                mask_samples[start:i] *= fade
            else:
                # Fade out of silence (falling mask edge)
                end = min(len(mask_samples), i + fade_samples)
                fade = np.linspace(0.0, 1.0, end - i)
                mask_samples[i:end] *= fade

    return mask_samples


def synthesize_wavegrad_masked(
    mel: np.ndarray,
    model,
) -> np.ndarray:
    """Synthesize via WaveGrad then apply silence mask to remove hum artifacts."""
    with torch.no_grad():
        waveform = model.forward(
            torch.from_numpy(mel).unsqueeze(0).float()
        ).squeeze(0).numpy()

    mask = build_silence_mask(mel, n_samples=len(waveform))
    return waveform * mask


# ---- Results after silence masking fix ----
#
# Test set: 50 utterances with long silent segments (>0.5s, em-dash pauses
# and long comma pauses) + 50 control utterances (standard prose, no long silences)
#
# Fix validation on silent-segment test set:
#   Before fix:
#     WaveGrad RMS during silence: -38 to -44 dBFS (hum audible)
#     FFT peak at 60-80 Hz: -36 to -42 dBFS
#     Hum rate (>3 dB above noise floor): 50/50 (100%)
#
#   After fix:
#     WaveGrad RMS during silence: < -72 dBFS (clean silence)
#     FFT peak at 60-80 Hz: < -70 dBFS (below noise floor)
#     Hum rate: 0/50 (0%)
#
# Control set (no long silences, 50 utterances):
#   Before fix: MOS 3.5 (σ=0.4)
#   After fix:  MOS 4.0 (σ=0.3)  ← improvement from hum elimination
#   Mask boundary clicks: 0 (fade-in/fade-out eliminates boundary discontinuity)
#
# MOS on utterances with long silences:
#   Before fix: MOS 3.5 (σ=0.5) — "low hum in pauses, distracting"
#   After fix:  MOS 4.0 (σ=0.3) — "clean pauses, natural rhythm"
#
# Minimum silent region for audible hum: ~0.3s (26 mel frames)
# Minimum mask threshold for clean silence: -60 dBFS mean log-mel energy
# Fade samples for click-free boundaries: 64 samples (2.9 ms at 22050 Hz)

print("Fix: silence mask hard-zero WaveGrad output at near-zero mel frames")
print("Hum rate: 100% → 0%; MOS: 3.5 → 4.0; 1.5h")

Related retainer posts

Track retainer hours without a client portal

HourTab turns your time-tracker CSV into a public URL your client bookmarks. No login. No portal. CSV in, URL out.

See pricing →