SpeechTokenizer developer on retainer: model.encode(audio) returns (n_q=8, T) (RVQ-layers-first) — TTS decoder expects (T, n_q) (time-first) — codes fed without .transpose(0, 1) — decoder receives RVQ layer sequences as frame sequences — MOS 1.8 — 3 weeks — SpeechTokenizer developer on monthly retainer

SpeechTokenizer v1.0.x • Python 3.11 • PyTorch 2.1.x • SpeechTokenizer.load_from_checkpoint('speechtokenizer_hubert_avg') • codes = model.encode(audio) → shape (n_q=8, T) • TTS decoder expects (T, n_q) time-first layout • codes fed to decoder without .transpose(0, 1) • decoder processes codes[0, :] as frame 0’s 8 RVQ codes → actually receives all T frames of RVQ layer 0 • generated audio: 8 frames of partial correct timbre → rapid wrong-layer token repetition • MOS 1.8 (expected 4.0+) • developer checks TTS decoder architecture, SpeechTokenizer model variant, audio preprocessing for 3 weeks • fix: codes = model.encode(audio).transpose(0, 1) • MOS 1.8→4.1 • 1.5h retainer engagement

Eastfield SpeechTokenizer TTS Ltd. was building an LLM-based text-to-speech system using SpeechTokenizer as the acoustic token representation. The TTS decoder was a GPT-style autoregressive transformer that generated speech token sequences frame by frame: at each step it predicted the 8 RVQ codes for the next audio frame, conditioned on the text and all previously generated frames. The expected decoder input layout was (T, n_q) — a 2D tensor where row t contained the 8 RVQ codes for audio frame t. The developer loaded the SpeechTokenizer model with model = SpeechTokenizer.load_from_checkpoint('speechtokenizer_hubert_avg') and encoded ground-truth audio for teacher-forced training with codes = model.encode(audio). The returned codes tensor had shape (8, T) where the first dimension was the 8 RVQ layers and the second dimension was the number of token frames. The developer fed codes directly to the decoder without transposing. The decoder received codes[0, :] as the codes for frame 0 — but codes[0, :] was the full T-frame sequence of RVQ layer 0 (the semantic content layer), not the 8 RVQ codes for the first audio frame. After training for 100k steps, generated audio showed 8 frames of partially correct timbre followed by rapid repetition of acoustically meaningless patterns. MOS on a 50-utterance evaluation set was 1.8, against an expected 4.0+ for SpeechTokenizer-based TTS. After three weeks checking the TTS decoder architecture, SpeechTokenizer checkpoint variant (speechtokenizer_hubert_avg vs speechtokenizer_snake), and audio preprocessing, the team engaged a SpeechTokenizer retainer developer. The root cause was a missing .transpose(0, 1) call between model.encode and the decoder input.

The SpeechTokenizer encoding and TTS training configuration at the time of the engagement:

# Eastfield SpeechTokenizer TTS Ltd. — LLM-based TTS training pipeline
# SpeechTokenizer v1.0.x, Python 3.11, PyTorch 2.1.x

import torch
from speechtokenizer import SpeechTokenizer

# ---- Load SpeechTokenizer ----
model = SpeechTokenizer.load_from_checkpoint('speechtokenizer_hubert_avg')
model.eval()
# SpeechTokenizer uses 8 RVQ layers (n_q=8).
# Layer 0 (first quantizer): captures semantic/content information
#   — correlated with HuBERT features, encodes phoneme identity
# Layers 1–7 (residual quantizers): capture acoustic details
#   — timbre, speaking style, background noise, fine-grained prosody

# ---- Encode ground-truth audio (teacher-forced training) ----
# audio: (B=1, 1, N_samples) at 16 kHz
audio = load_audio("training_utterance.wav")  # shape: (1, 1, 16000)

with torch.no_grad():
    codes = model.encode(audio)
    # codes shape: (n_q=8, T)
    # n_q=8: 8 RVQ quantization layers
    # T: number of token frames (= N_samples / hop_length, typically T ≈ 50 at 16 kHz)
    #
    # codes[0, :]: shape (T,) — all T frames from RVQ layer 0 (semantic layer)
    # codes[1, :]: shape (T,) — all T frames from RVQ layer 1 (first acoustic layer)
    # ...
    # codes[7, :]: shape (T,) — all T frames from RVQ layer 7

# ---- TTS decoder expects (T, n_q) time-first layout ----
# The GPT-style decoder processes the token sequence frame by frame:
# - Input at step t: codes_t = codes[t, :] → shape (n_q,) — the 8 RVQ codes for frame t
# - The decoder predicts codes[t+1, :] from codes[0:t+1, :] and the text conditioning
#
# CORRECT layout (T, n_q):
# codes[t, :] → shape (8,): [layer_0_code_at_t, layer_1_code_at_t, ..., layer_7_code_at_t]
# codes[0, :] = 8 RVQ codes for audio frame 0
# codes[1, :] = 8 RVQ codes for audio frame 1
# ...
# codes[T-1, :] = 8 RVQ codes for audio frame T-1

# ---- Bug: feeding codes directly without transpose ----
# codes shape: (8, T) — model.encode output (RVQ-layers-first)
# decoder_input = codes  ← BUG: decoder receives (8, T), treats as (T=8, n_q=T)
#
# From the decoder's perspective:
# "frame 0's codes" = codes[0, :] = shape (T,)
#   → decoder reads T values as the 8 RVQ codes for frame 0
#   → actual content: all T frames of semantic layer 0's tokens
# "frame 1's codes" = codes[1, :] = shape (T,)
#   → actual content: all T frames of acoustic layer 1's tokens
# "frame 7's codes" = codes[7, :] = shape (T,)
#   → actual content: all T frames of acoustic layer 7's tokens
#
# The decoder is trained to generate an 8-frame token sequence
# (n_q=8 "frames", each with T "codes" per "layer")
# This is semantically nonsensical: the model learns to predict
# complete layer-0-sequence → complete layer-1-sequence → ... → complete layer-7-sequence,
# rather than the actual acoustic structure of frame-by-frame generation.
#
# Generated audio result:
# - First ~8 frames (100–160 ms): acoustic content loosely resembles speaker timbre
#   (decoder has seen the semantic layer token distribution during training)
# - Remaining frames: rapid repetition of wrong-layer token patterns
# - MOS: 1.8 (expected 4.0+ from SpeechTokenizer-based TTS)
#
# Developer checks for 3 weeks:
# 1. TTS decoder hidden size (256 → 512 → 1024): no improvement
# 2. SpeechTokenizer checkpoint (hubert_avg → snake): no improvement
# 3. Audio preprocessing (normalization, resampling): no improvement
# 4. Training duration (50k → 100k → 200k steps): slight improvement to MOS 2.1
# Never checks the axis order of model.encode output.

# ---- Roundtrip test (not done by developer, would have revealed the issue) ----
# Encoding and immediately decoding should reproduce the original audio:
reconstructed = model.decode(codes)  # codes shape: (8, T) ← model.decode accepts (n_q, T)
# MOS of reconstructed vs original: ~3.9 (SpeechTokenizer self-reconstruction quality)
# → If developer had run this test and confirmed MOS ~3.9, then fed codes to decoder
#   and got MOS 1.8, the problem would be isolated to the decoder interface, not the codec.
# Developer did not run the roundtrip test.

Why the (n_q, T) → (T, n_q) transpose matters for frame-by-frame TTS decoders

SpeechTokenizer encodes audio into a residual vector quantization (RVQ) representation: the input waveform is first encoded into a continuous latent vector, then quantized through 8 successive RVQ layers. Each quantizer layer produces one discrete code per audio frame; with 8 layers and T frames, the full representation is a matrix of 8 × T codes. The model.encode API returns this matrix in layer-first order — shape (n_q, T) — where the first axis indexes the RVQ layer (0 = semantic, 1–7 = acoustic details) and the second axis indexes the time frame. This layout is natural for codec-internal operations that process one RVQ layer at a time.

Autoregressive TTS decoders that model the speech token sequence as a language modeling problem operate in time-first order: at each decoding step t, the model receives the codes for frame t (all 8 RVQ codes for that moment in time) as a single input vector, and predicts the codes for frame t+1. The model is trained to learn the statistical dependencies between consecutive frames in time, using the 8 RVQ codes as a joint acoustic descriptor of each time step. For this architecture, the decoder input must be in (T, n_q) layout: row t must contain the 8 RVQ codes for frame t.

When codes in (n_q=8, T) layout is fed to a (T, n_q) decoder, the decoder receives 8 “frames” (the value of n_q), each with T “codes”. It interprets codes[0, :] — the full semantic-layer token sequence — as the multi-code descriptor for frame 0. Training on this wrong layout causes the decoder to learn to predict complete per-layer token streams rather than per-frame acoustic snapshots: a fundamentally different and much harder prediction problem. The model partially succeeds at approximating the statistical distribution of the semantic layer’s tokens, producing a brief period of loosely plausible timbre in the generated audio, before collapsing to incoherent patterns from the wrong-layer interleaving. The MOS of 1.8 reflects this partial coherence: the audio is not random noise (which would score ~1.0–1.2 on MOS) but is not intelligible speech either.

# Fix: add .transpose(0, 1) between model.encode and the TTS decoder
# SpeechTokenizer v1.0.x, Python 3.11, PyTorch 2.1.x

import torch
from speechtokenizer import SpeechTokenizer

model = SpeechTokenizer.load_from_checkpoint('speechtokenizer_hubert_avg')
model.eval()

audio = load_audio("training_utterance.wav")  # (1, 1, N_samples) at 16 kHz

with torch.no_grad():
    codes_nq_T = model.encode(audio)
    # codes_nq_T shape: (n_q=8, T) — SpeechTokenizer's native (layers-first) layout

    # ---- Fix: transpose to (T, n_q) for time-first TTS decoder ----
    codes_T_nq = codes_nq_T.transpose(0, 1)
    # codes_T_nq shape: (T, n_q=8) — time-first layout for autoregressive decoder
    #
    # Now: codes_T_nq[t, :] = shape (8,) — 8 RVQ codes for audio frame t
    # codes_T_nq[0, :] = [layer_0_code_frame_0, layer_1_code_frame_0, ..., layer_7_code_frame_0]
    # codes_T_nq[1, :] = [layer_0_code_frame_1, layer_1_code_frame_1, ..., layer_7_code_frame_1]
    # ...

    # ---- Add a shape assertion to catch future axis confusion ----
    n_q = model.quantizer.n_q   # = 8
    assert codes_T_nq.shape[1] == n_q, (
        f"Expected codes last dim = n_q={n_q}, got {codes_T_nq.shape[1]}. "
        f"Did you forget .transpose(0, 1) on model.encode output?"
    )

# ---- Feed transposed codes to TTS decoder ----
decoder_input = codes_T_nq  # shape: (T, 8) ← correct for time-first decoder

# ---- Verify with roundtrip test ----
# model.decode expects (n_q, T) — the original layout, not transposed
codes_for_decode = codes_T_nq.transpose(0, 1)  # (T, n_q) → (n_q, T) for decode
reconstructed = model.decode(codes_for_decode)
# MOS of reconstructed vs original: ~3.9 (SpeechTokenizer self-reconstruction quality)
# If roundtrip MOS ≥ 3.5, the codec is working correctly.
# Any lower roundtrip MOS indicates a codec-level issue (checkpoint mismatch, sample rate mismatch).

# ---- Results after fix ----
# TTS model retrained from scratch with correct (T, n_q) decoder input layout:
# MOS on 50-utterance evaluation set: 4.1 (up from 1.8)
# UTMOS: 3.9 (up from 1.6)
# Listening test: generated speech intelligible, speaker identity preserved,
#   prosody natural — consistent with SpeechTokenizer-based TTS quality targets
#
# ---- Note on model.decode layout ----
# model.decode(codes) expects codes in (n_q, T) layout (same as model.encode output).
# If you store codes in (T, n_q) layout (after transpose), you must un-transpose
# before passing to model.decode: model.decode(codes_T_nq.transpose(0, 1))
#
# ---- Hour breakdown ----
# Triage:     0.25h — MOS 1.8; first 8 frames loosely correct then artifacts;
#             roundtrip encode→decode MOS 3.9 (codec fine);
#             fault is between codec output and decoder input
# Root cause: 0.5h — model.encode returns (n_q=8, T); decoder expects (T, n_q);
#             codes[i,:] = all T frames of layer i, not 8 codes for frame i;
#             axis transposition missing
# Fix:        0.5h — codes = model.encode(audio).transpose(0, 1);
#             shape assertion added; roundtrip test documented
# Validation: 0.25h — MOS 1.8→4.1; UTMOS 1.6→3.9; 1.5h total

Related retainer posts

Track retainer hours without a client portal

HourTab turns your time-tracker CSV into a public URL your client bookmarks. No login. No portal. CSV in, URL out.

See pricing →