LPCNet developer on retainer: whispered speech synthesized correctly for first 400 ms then reverts to fully voiced speech for remainder of utterance — GRU hidden state initialized to zero converges toward voiced training-distribution mean after 400 ms — initializing GRU state with mean whispered-speech state vector fixes whisper persistence — 3 weeks — LPCNet developer on monthly retainer

LPCNet (mozilla/LPCNet) traditional LPC + neural vocoder hybrid • Python 3.11; C extension • real-time low-bandwidth TTS pipeline • whispered speech input (text prefixed with [WHISPER] control token) synthesized with correct whisper phoneme quality for first 400 ms then transitions to fully voiced speech for remainder of utterance — whisper effect does not persist beyond 400 ms • normal voiced speech synthesizes correctly • whisper-only short inputs under 400 ms are fully correct • developer checks LPCNet configuration, feature input pipeline, and excitation model for 3 weeks • root cause: LPCNet GRU hidden state initialized to zero converges toward training-distribution mean dominated by voiced speech (95%+ voiced training corpus); after approximately 400 ms GRU convergence time, excitation prediction reverts to voiced; fix: initialize GRU hidden state with mean whispered-speech state vector extracted from whispered-speech training examples when input is flagged as whispered • whisper persistence: 400 ms → full utterance duration • MOS: 3.5 → 3.9 • 1.5h retainer engagement

Northfield LPCNet Ltd. built a real-time low-bandwidth TTS pipeline using LPCNet for a voice interface that required multiple synthesis modes, including a whispered speech mode for quiet-environment use cases. The pipeline used a text-to-features model that accepted control tokens in the input text: prepending [WHISPER] to an input sentence was intended to signal the downstream vocoder to synthesize in whispered mode. For short utterances under 400 ms (a few words), the whisper mode worked correctly: the output had the correct unvoiced, breathy quality of natural whispered speech. But for any utterance longer than 400 ms — which included most complete sentences — the synthesis began correctly in whisper mode and then, at approximately the 400 ms mark, transitioned abruptly into fully voiced modal speech for the remainder of the utterance. The transition was clearly audible: the first few words were whispered, then the rest of the sentence switched to normal voiced speech with no whisper quality.

The team initially checked whether the control token was being correctly passed through the text-to-features pipeline and whether the acoustic features it generated for whispered speech were correctly representing the voicing information across the full utterance. The features looked correct frame-by-frame; the whisper-mode features were maintained throughout the utterance. They then checked whether a frame-count-dependent threshold in the LPCNet C extension was capping whisper mode, and whether a post-processing step was inadvertently re-voicing the signal. Neither was present. After three weeks they engaged an LPCNet retainer developer who identified the root cause as the GRU hidden state’s convergence toward the voiced-speech training distribution mean after approximately 40 frames of zero initialization.

The LPCNet inference pipeline and GRU state diagnostic at the time of the engagement:

# Northfield LPCNet Ltd. — LPCNet whisper persistence failure
# LPCNet (mozilla/LPCNet) traditional LPC + neural vocoder hybrid
# Python 3.11, C extension
# Deployment: real-time low-bandwidth TTS pipeline with whisper mode

import ctypes
import numpy as np

# Load LPCNet shared library (compiled from mozilla/LPCNet)
lpcnet_lib = ctypes.CDLL('./liblpcnet.so')

# LPCNet inference: Python wrapper around LPCNet's C inference loop
# lpcnet_synthesize(features, gru_state_in) → (audio_frames, gru_state_out)
# features: (T, 55) — LPCNet acoustic features, 10 ms per frame
# gru_state_in: (1, GRU_SIZE) — GRU hidden state at utterance start
# Returns: (audio_samples, gru_state_out) — 160 audio samples per 10 ms frame

GRU_SIZE = 384  # LPCNet GRU hidden state dimension

# --- DIAGNOSTIC: GRU state trajectory during whisper utterance ---
# Load whispered-speech features for "She whispered very quietly to him"
# (long utterance: ~1.8 seconds = 180 frames at 10 ms per frame)
features_whisper = load_lpcnet_features("[WHISPER] She whispered very quietly to him")
# features_whisper shape: (180, 55)

# Synthesize with zero GRU initialization (default — buggy)
gru_state = np.zeros((1, GRU_SIZE), dtype=np.float32)
audio_frames = []
gru_states_over_time = []

for t in range(len(features_whisper)):
    frame_features = features_whisper[t:t+1]  # (1, 55)
    audio_frame, gru_state = lpcnet_synthesize_frame(frame_features, gru_state)
    audio_frames.append(audio_frame)
    gru_states_over_time.append(gru_state.copy())

# Measure voicing (zero-crossing rate) per frame
# High ZCR → unvoiced/whispered; Low ZCR → voiced/periodic
audio_all = np.concatenate(audio_frames)
frame_samples = 160  # 10 ms at 16000 Hz

voiced_frames = []
for t in range(len(audio_frames)):
    frame = audio_frames[t]
    zcr = np.sum(np.diff(np.sign(frame)) != 0) / len(frame)
    is_voiced = zcr < 0.15  # low ZCR = voiced periodic signal
    voiced_frames.append(is_voiced)

# Find frame at which whisper transitions to voiced
transition_frame = next((t for t in range(len(voiced_frames))
                         if voiced_frames[t] and t > 5), None)
print(f"Whisper-to-voiced transition at frame {transition_frame} = {transition_frame * 10} ms")
# Output: Whisper-to-voiced transition at frame 42 = 420 ms

# GRU state at frame 0 (near-zero): L2 norm relative to voiced-speech mean
voiced_mean_state = load_precomputed_voiced_mean_gru_state()  # precomputed reference
print(f"GRU state at frame 0:  L2 distance from voiced mean = {np.linalg.norm(gru_states_over_time[0] - voiced_mean_state):.3f}")
# Output: GRU state at frame 0:  L2 distance from voiced mean = 12.847

print(f"GRU state at frame 42: L2 distance from voiced mean = {np.linalg.norm(gru_states_over_time[42] - voiced_mean_state):.3f}")
# Output: GRU state at frame 42: L2 distance from voiced mean = 1.203 ← converged

Why the GRU hidden state converges toward the voiced training distribution mean and how whispered-speech state initialization prevents it

LPCNet’s neural component is a GRU-based network that learns to predict the residual excitation signal for the LPC synthesis filter. At each 10 ms frame, the GRU takes the current acoustic feature vector and its previous hidden state as input and outputs an excitation prediction. The network was trained on a corpus that is approximately 95% modal voiced speech, with whispered speech comprising approximately 5% of the training data. This corpus imbalance means that the attractor basin of the GRU’s hidden state dynamics is strongly concentrated in the region corresponding to voiced-speech excitation patterns. The voiced-speech attractor has a large basin; the whispered-speech attractor has a small one.

When the GRU state is initialized to zero at utterance start, it lies far from both the voiced and whispered attractors. For the first several frames, the acoustic features (which carry whisper-mode information) are strong enough to drive the GRU state toward the whispered-speech region, overcoming the network’s learned bias toward the voiced attractor. But as the GRU state accumulates across frames, the dynamics of the system pull it toward the deeper voiced attractor. After approximately 40 frames (400 ms), the GRU state has crossed the boundary between the whispered and voiced basins and converges into the voiced-speech region, at which point the excitation prediction switches to voiced regardless of the acoustic feature input. The fix pre-computes the mean GRU state extracted from inference on whispered-speech training examples — the centroid of the whispered-speech attractor basin — and initializes the GRU from this state when the input is whispered. Starting inside the whispered attractor basin, the GRU state dynamics keep the network in whispered-speech mode for the duration of the utterance.

# Fix: initialize LPCNet GRU hidden state with mean whispered-speech state vector
# when input is flagged as whispered
# State vectors pre-extracted offline from whispered-speech training examples

import numpy as np
import ctypes

lpcnet_lib = ctypes.CDLL('./liblpcnet.so')
GRU_SIZE = 384

# --- Offline step: extract mean whispered-speech GRU state ---
# (run once during deployment setup, save to disk)
def extract_whisper_mean_gru_state(whisper_training_examples: list) -> np.ndarray:
    """
    Run LPCNet inference on a set of whispered-speech training examples,
    collect GRU states at steady state (after first 300 ms), and return mean.
    """
    steady_state_vectors = []
    zero_init = np.zeros((1, GRU_SIZE), dtype=np.float32)

    for features in whisper_training_examples:
        # features: (T, 55) whispered-speech LPCNet features
        gru_state = zero_init.copy()
        for t in range(len(features)):
            _, gru_state = lpcnet_synthesize_frame(features[t:t+1], gru_state)
            if t >= 30:  # collect state from frame 30+ (300 ms, past transient)
                steady_state_vectors.append(gru_state.copy())

    return np.mean(steady_state_vectors, axis=0)

# Load pre-computed mean whisper GRU state (extracted during deployment setup)
WHISPER_MEAN_GRU_STATE = np.load('lpcnet_whisper_mean_gru_state.npy')
# shape: (1, 384)

# --- Inference with mode-specific GRU initialization ---
def synthesize_with_mode_init(features: np.ndarray, is_whispered: bool) -> np.ndarray:
    """
    Synthesize LPCNet audio with GRU state initialized for the target voice mode.
    """
    if is_whispered:
        gru_state = WHISPER_MEAN_GRU_STATE.copy()
    else:
        gru_state = np.zeros((1, GRU_SIZE), dtype=np.float32)

    audio_frames = []
    for t in range(len(features)):
        audio_frame, gru_state = lpcnet_synthesize_frame(features[t:t+1], gru_state)
        audio_frames.append(audio_frame)

    return np.concatenate(audio_frames)


# ---- Results after whispered-speech GRU state initialization fix ----
#
# Test set: 100 whispered-speech utterances from 0.5 to 5.0 seconds duration
# Voicing test: frame classified as voiced if ZCR < 0.15 (periodic excitation)
#
# Whisper persistence (% frames correctly classified as unvoiced/whispered):
#   Buggy (zero init):   Frames 0–40: 97% unvoiced; Frames 41+: 8% unvoiced (reverted)
#   Fixed (whisper init): Frames 0–40: 98% unvoiced; Frames 41+: 95% unvoiced (persists)
#
# MOS evaluation (20-listener crowdsource, 50 whispered utterances 1–3 s duration):
#   Buggy:  MOS 3.5 (σ=0.6) — "whisper fades into normal speech partway through"
#   Fixed:  MOS 3.9 (σ=0.4) — "clear whisper quality throughout"
#
# Normal voiced speech (control — zero init unchanged):
#   Before fix: MOS 4.1  →  After fix: MOS 4.1 (no regression — zero init preserved)
#
# Short utterances under 400 ms (control — zero init already works):
#   Before fix: MOS 3.9  →  After fix: MOS 3.9 (no change, as expected)

print("Fix: LPCNet GRU state initialized from mean whispered-speech state vector")
print("Whisper persistence: 400 ms → full utterance duration; MOS: 3.5 → 3.9; 1.5h")

Related retainer posts

Track retainer hours without a client portal

HourTab turns your time-tracker CSV into a public URL your client bookmarks. No login. No portal. CSV in, URL out.

See pricing →