EnCodec developer on retainer: codebook-0 swap voice conversion ignores RVQ residual structure — residual codes 1–7 override swapped codebook within 15 frames — ECAPA-TDNN speaker similarity 0.21 — 3 weeks — EnCodec developer on monthly retainer

EnCodec 0.1.2 • Python 3.11 • PyTorch 2.1.x • EncodecModel.encodec_model_24khz() • model.set_target_bandwidth(6.0) → n_q=8 codebooks • source-speaker encoded_frames with codebook 0 replaced by target-speaker codebook 0 • decoded output matches target identity in first 10–15 frames (~30–45 ms) → abruptly reverts to source speaker • ECAPA-TDNN cosine similarity 0.21 (expected 0.70+) • developer checks bandwidth (3.0/6.0/12.0 kbps), EnCodec model (24 kHz vs 48 kHz), audio preprocessing for 3 weeks • fix: replace codebook-swap approach with kNN-VC (HuBERT self-supervised features + k-nearest-neighbour unit conversion) • ECAPA-TDNN similarity 0.21→0.72 • 1.5h retainer engagement

Westfield EnCodec Audio Ltd. was building a voice conversion system using Meta’s EnCodec neural audio codec. The plan was to encode a source-speaker utterance into EnCodec discrete codes, encode a reference target-speaker utterance, replace codebook 0 of the source frames with codebook 0 of the target frames, and decode the modified frames to produce a voice-converted output. The reasoning was that codebook 0 — the first and coarsest RVQ quantizer — should capture high-level spectral structure including speaker identity, while residual codebooks 1–7 would encode finer acoustic details like background noise and room acoustics. The developer loaded the model with model = EncodecModel.encodec_model_24khz(), set model.set_target_bandwidth(6.0) (activating n_q=8 codebooks at 6 kbps), encoded both utterances via model.encode(), replaced the first codebook in each encoded frame tuple, and decoded with model.decode().

The decoded output was disappointing: the first 10–15 audio frames (~30–45 ms) sounded loosely like the target speaker, but by frame 20 the audio had completely reverted to source-speaker characteristics and remained there for the full utterance duration. The developer measured ECAPA-TDNN speaker embedding cosine similarity between the output and the target speaker reference at 0.21 — barely above what a random speaker pair scores (0.0) and far below the 0.70+ threshold for successful voice conversion. After three weeks checking bandwidth settings (3.0, 6.0, 12.0 kbps), the EnCodec model variant (24 kHz versus 48 kHz), and audio preprocessing (normalization, resampling), the team engaged an EnCodec retainer developer. The root cause was a fundamental incompatibility between codebook-swap voice conversion and EnCodec’s residual vector quantization architecture.

The EnCodec encoding configuration at the time of the engagement:

# Westfield EnCodec Audio Ltd. — voice conversion pipeline
# EnCodec 0.1.2, Python 3.11, PyTorch 2.1.x

import torch
from encodec import EncodecModel
from encodec.utils import convert_audio

# ---- Load EnCodec model ----
model = EncodecModel.encodec_model_24khz()
model.set_target_bandwidth(6.0)
# 6.0 kbps at 24 kHz → n_q=8 active RVQ codebooks
# Each codebook: 1024 entries, 128-dim vectors
# Total: 8 × 75 codes/sec = 600 codes/sec at 24 kHz

model.eval()

# ---- Encode source speaker utterance ----
source_wav = load_and_resample("source_speaker.wav", 24000)   # (1, 1, N_samples)
target_wav = load_and_resample("target_reference.wav", 24000) # (1, 1, N_ref)

with torch.no_grad():
    source_encoded = model.encode(source_wav)
    target_encoded = model.encode(target_wav)
    # encoded: list of EncodedFrame tuples
    # Each EncodedFrame: (codes, scale)
    # codes shape: (B=1, n_q=8, T)  — B=batch, n_q=codebooks, T=time frames
    # scale shape: (B=1, 1) — per-frame scale factor for lossless normalization

# ---- Codebook-0 swap: replace source codebook 0 with target codebook 0 ----
# Developer's hypothesis: codebook 0 = coarse speaker identity
# Residual codebooks 1–7 = fine acoustic details (speaker-independent)
modified_frames = []
for (src_codes, src_scale), (tgt_codes, _) in zip(source_encoded, target_encoded):
    # src_codes shape: (1, 8, T_src)
    # tgt_codes shape: (1, 8, T_tgt) — may differ in length from source
    # Swap codebook 0 (first RVQ layer) from target into source
    modified_codes = src_codes.clone()
    T_min = min(src_codes.shape[2], tgt_codes.shape[2])
    modified_codes[:, 0, :T_min] = tgt_codes[:, 0, :T_min]
    # modified_codes[:, 0, :]: target-speaker codebook-0 codes
    # modified_codes[:, 1:, :]: source-speaker residual codes 1–7
    modified_frames.append((modified_codes, src_scale))

# ---- Decode modified frames ----
with torch.no_grad():
    decoded_vc = model.decode(modified_frames)
    # decoded_vc: (1, 1, N_samples) waveform

# ---- Measure speaker similarity ----
# ECAPA-TDNN cosine similarity between decoded_vc and target_reference:
# similarity = 0.21 (expected 0.70+ for successful voice conversion)
#
# Listening test: first ~15 frames (~40 ms) loosely resemble target speaker timbre;
# from frame 16 onward: source speaker characteristics dominate completely
#
# Developer checks for 3 weeks:
# 1. Bandwidth setting (3.0, 6.0, 12.0 kbps): n_q=4, 8, 16 — similarity 0.19–0.23
# 2. Model variant (encodec_model_48khz): similarity 0.20
# 3. Audio preprocessing (peak normalize, resample 44.1→24 kHz): similarity 0.22
# 4. Swapping codebooks 0+1 together: similarity 0.26 (slight improvement, still fails)
# Never investigates why residual codes 1–7 override the codebook-0 swap.

Why EnCodec’s RVQ residual structure defeats codebook-swap voice conversion

EnCodec encodes audio by first passing the waveform through a strided convolutional encoder into a continuous latent vector, then quantizing that vector through 8 successive residual VQ layers. Each RVQ layer quantizes the residual error left by all previous layers: codebook 0 finds the nearest entry to the latent vector; codebook 1 quantizes the error between the latent and codebook-0’s reconstruction; codebook 2 quantizes the residual error after codebooks 0 and 1; and so on. The final decoded waveform is the sum of all 8 codebook vectors mapped back through the decoder. No codebook independently encodes a semantically meaningful attribute like speaker identity.

The critical consequence for voice conversion: codebook 0’s entries are optimized to cover the distribution of encoder latent vectors from many speakers, not to tile the space of speaker identities. The residual codes 1–7 were computed specifically to correct the residual error relative to the source speaker’s codebook-0 selection. When the developer replaces codebook 0 with the target-speaker’s selection, the residual codes 1–7 no longer correct the target-speaker latent — they correct the source-speaker latent minus source-speaker codebook 0. During decoding, the target-speaker codebook-0 vector contributes a slightly target-shaped spectral envelope in the first few frames, but the source-speaker residual vectors progressively add back the source-speaker’s characteristic harmonics, formant positions, and prosodic contours. By frame 15–20, the residual sum has fully reconstructed the source-speaker’s spectral fingerprint, and the decoded audio is perceptually indistinguishable from the unmodified source speaker.

# Fix: replace codebook-swap approach with kNN-VC (HuBERT + k-NN unit matching)
# EnCodec 0.1.2, Python 3.11, PyTorch 2.1.x

# kNN-VC uses HuBERT self-supervised features (speaker-independent phoneme representations)
# for voice conversion, then optionally uses EnCodec as a pure audio codec for
# bandwidth-efficient output transmission — not as the conversion mechanism itself.

import torch
import torchaudio
from knnvc import KNN_VC  # pip install knn-vc

# ---- Load kNN-VC model (uses WavLM-Large prematched features) ----
knnvc = KNN_VC(prematched=True, device='cuda')
# kNN-VC extracts self-supervised speech representations from WavLM-Large
# that are more speaker-independent than EnCodec codebook 0:
# - WavLM-Large layer 6 features correlate strongly with phoneme identity
# - WavLM-Large layer 6 features are substantially less correlated with
#   speaker identity than EnCodec codebook 0 entries

# ---- Build matching set from target speaker reference audio ----
target_wavs = ["target_ref_1.wav", "target_ref_2.wav", "target_ref_3.wav"]
# More target-speaker utterances → better k-NN coverage → better voice quality
matching_set = knnvc.get_matching_set(target_wavs)
# matching_set: (N_frames, 1024) — WavLM features from all target-speaker frames

# ---- Convert source speaker to target speaker voice ----
source_wav, sr = torchaudio.load("source_speaker.wav")
query_seq = knnvc.get_features(source_wav)
# query_seq: (T, 1024) — WavLM features of source-speaker utterance

out_wav = knnvc.match(query_seq, matching_set, topk=4)
# match(): for each source frame, find top-k nearest WavLM-feature frames
# in the target matching set; replace source frame with weighted average
# of top-k target frames; vocode the result through HiFi-GAN
# out_wav: (1, N_samples) — target-voice waveform

# ---- Results after replacing codebook-swap with kNN-VC ----
# ECAPA-TDNN cosine similarity (output vs target reference):
#   Before (codebook-0 swap): 0.21
#   After (kNN-VC):           0.72
#
# MOS naturalness (50-utterance eval set):
#   kNN-VC output: 3.6 MOS (down from source speaker 4.2, expected for any VC system)
#
# EnCodec retained as audio codec: kNN-VC output optionally compressed to 6 kbps
# via model.encode(out_wav) for bandwidth-constrained transmission — codec reconstruction
# SI-SDR 14.3 dB (EnCodec compression quality unaffected by upstream VC change)
#
# ---- Why kNN-VC works where codebook-0 swap fails ----
# kNN-VC self-supervised features (WavLM-Large layer 6):
# - Encode phoneme identity (high correlation with phone boundaries)
# - Have substantially less speaker-identity information than raw waveform
# - Are not entangled with the specific residual quantization of the source speaker
#
# EnCodec RVQ codebooks:
# - Encode the full waveform reconstruction signal including all speaker characteristics
# - Are entangled: codebook i is defined relative to the residual error of codebooks 0..i-1
# - Cannot be swapped independently to transfer speaker identity
#
# ---- Hour breakdown ----
# Triage:     0.25h — ECAPA-TDNN 0.21; first 15 frames loosely target-like;
#             codec self-reconstruction SI-SDR 14.2 dB (codec itself is fine);
#             fault is the voice conversion architecture, not the codec
# Root cause: 0.5h — EnCodec RVQ residual structure; codebook 0 not speaker-disentangled;
#             residual codes 1–7 computed relative to source-speaker codebook-0 vector;
#             override target-speaker codebook-0 within 15 frames; approach incompatible
# Fix:        0.5h — kNN-VC (WavLM-Large features; k-NN matching set from target speaker;
#             HiFi-GAN vocoder); EnCodec retained as pure audio codec for output compression
# Validation: 0.25h — ECAPA-TDNN 0.21→0.72; MOS 3.6; 1.5h total

Related retainer posts

Track retainer hours without a client portal

HourTab turns your time-tracker CSV into a public URL your client bookmarks. No login. No portal. CSV in, URL out.

See pricing →