Lhotse developer on retainer: MP3-format VoxPopuli files in CutSet trigger Sox subprocess per file — 16 DataLoader workers spawn 16 concurrent Sox processes saturating SATA SSD disk I/O — DataLoader workers IO-block waiting for disk — GPU utilization 23% vs expected 85% — 3 weeks — Lhotse developer on monthly retainer

lhotse 1.19.x • Python 3.11 • PyTorch 2.1.x • large-scale ASR training on 10K-hour multi-corpus CutSet (LibriSpeech FLAC + TED-LIUM-3 FLAC + VoxPopuli MP3) • BucketingSampler(max_duration=300, num_buckets=20) • DataLoader(num_workers=16) • MP3 VoxPopuli files dispatch to Sox subprocess backend (not soundfile) • 16 concurrent Sox subprocesses + SATA SSD at 350 MB/s • disk I/O saturated at 95%+ • DataLoader workers IO-block • GPU utilization drops to 23% (expected 85%+) • developer checks num_workers (8, 16, 32), pin_memory, BucketingSampler.max_duration for 3 weeks • fix: convert VoxPopuli MP3s to FLAC offline • soundfile decode 8× faster than Sox subprocess • GPU utilization 23%→84% • 1.5h retainer engagement

Eastfield Lhotse Corpus Ltd. was training a large Conformer-CTC ASR model on a 10,000-hour multi-corpus dataset assembled from LibriSpeech (960 hours, FLAC format), TED-LIUM-3 (452 hours, FLAC format), and VoxPopuli (en, 543 hours, MP3 format). The developer built a CutSet.from_manifests(recordings, supervisions) combining all three corpora and used a BucketingSampler(cuts, max_duration=300, num_buckets=20) for bucketed batching. Training on a machine with 2×A100 GPUs, 32 CPU cores, and a SATA SSD (350 MB/s read throughput) achieved only 23% average GPU utilization, with training throughput at 1,850 audio-seconds per second instead of the expected 6,000+. After three weeks of adjusting num_workers between 8, 16, and 32, toggling pin_memory, and varying the BucketingSampler.max_duration parameter, the team engaged a Lhotse retainer developer. The MP3 format of the VoxPopuli files was triggering Lhotse to dispatch audio loading through a Sox subprocess instead of the soundfile C extension, and 16 concurrent Sox subprocesses were saturating the SATA SSD bus.

The data loading pipeline at the time of the engagement:

# Eastfield Lhotse Corpus Ltd. — 10K-hour multi-corpus ASR training
# lhotse 1.19.x, Python 3.11, PyTorch 2.1.x

from lhotse import CutSet, RecordingSet, SupervisionSet
from lhotse.dataset import K2SpeechRecognitionDataset
from lhotse.dataset.sampling import BucketingSampler
from torch.utils.data import DataLoader
import torch

# Load corpora manifests
librispeech_recs  = RecordingSet.from_jsonl("librispeech_recordings.jsonl.gz")   # FLAC
tedlium_recs      = RecordingSet.from_jsonl("tedlium_recordings.jsonl.gz")        # FLAC
voxpopuli_recs    = RecordingSet.from_jsonl("voxpopuli_recordings.jsonl.gz")      # MP3

# ← Problem: CutSet combines FLAC corpora (soundfile backend) with
# MP3 VoxPopuli (Sox subprocess backend) transparently.
# Lhotse's Recording.load_audio() dispatches based on file extension:
#
#   .flac → soundfile.read(path)
#         → Python C extension, reads file directly into numpy array
#         → 1 file read (sequential, no subprocess)
#         → typical latency: 5-15ms per 30-second FLAC file
#         → I/O pattern: 1 disk read per file, ~1-3 MB
#
#   .mp3 → subprocess.Popen(['sox', path, '-t', 'raw', '-e', 'signed',
#                             '-b', '16', '-r', '16000', '-'])
#         → Sox subprocess: fork + exec + MP3 decode + write to pipe
#         → 1 subprocess spawn + 2 disk reads (MP3 header + data) + pipe read
#         → typical latency: 50-200ms per 30-second MP3 file
#         → I/O pattern: 2-3 disk reads per file, ~1.5-4 MB total
#
# CutSet does NOT normalize audio formats — it uses the backend
# determined by each Recording's source file extension.

librispeech_sups  = SupervisionSet.from_jsonl("librispeech_supervisions.jsonl.gz")
tedlium_sups      = SupervisionSet.from_jsonl("tedlium_supervisions.jsonl.gz")
voxpopuli_sups    = SupervisionSet.from_jsonl("voxpopuli_supervisions.jsonl.gz")

cuts = CutSet.from_manifests(
    recordings=librispeech_recs + tedlium_recs + voxpopuli_recs,
    supervisions=librispeech_sups + tedlium_sups + voxpopuli_sups,
)
# CutSet contains 285,000 cuts from LibriSpeech + TED-LIUM + VoxPopuli.
# Format distribution:
#   LibriSpeech: 132,553 cuts, 100% FLAC
#   TED-LIUM-3:   62,147 cuts, 100% FLAC
#   VoxPopuli:    90,448 cuts, 100% MP3  ← 31.7% of all cuts
# MP3 cuts processed by Sox subprocess backend.

sampler = BucketingSampler(
    cuts,
    max_duration=300,   # max 300s of audio per batch
    num_buckets=20,
    shuffle=True,
    buffer_size=10000,
)

dataset = K2SpeechRecognitionDataset(return_cuts=True)

# ← Problem: num_workers=16 means 16 DataLoader workers run concurrently.
# When a worker processes an MP3 cut, it spawns a Sox subprocess.
# With 16 workers, up to 16 Sox subprocesses run simultaneously.
#
# SATA SSD throughput: 350 MB/s read
# MP3 file I/O pattern per file (30-second clip at ~128kbps): ~0.5 MB audio data
# Sox subprocess overhead: 2-3 disk reads (file header, data, Sox binary load) = ~2 MB
# 16 concurrent Sox subprocesses disk I/O: 16 × (2 MB / 0.1s) = 320 MB/s
# SATA SSD saturation: 320 MB/s / 350 MB/s = 91% disk I/O utilization
#
# Result: each DataLoader worker waits 150-400ms for disk I/O instead of 5-15ms
# GPU waits for DataLoader workers to finish decoding the batch
# GPU utilization: 23% (8-second decode waits between 2-second GPU compute steps)
#
# Increasing num_workers to 32 makes it worse:
# 32 Sox subprocesses = 32 × 2 MB / 0.1s = 640 MB/s > 350 MB/s SSD throughput
# Workers compete for disk, further increasing decode latency

dataloader = DataLoader(
    dataset,
    batch_sampler=sampler,
    num_workers=16,         # ← 16 workers, 31.7% MP3 → Sox subprocess per MP3 file
    pin_memory=True,
    collate_fn=dataset.collate_fn,
)

# Training throughput:
# GPU utilization:         23% (expected 85%+)
# Audio-seconds per sec:   1,850 (expected 6,000+)
# Epoch duration (10K h):  ~54 hours (expected ~18 hours)

Why MP3 triggers Sox subprocess in Lhotse and how 16 concurrent Sox processes saturate a SATA SSD

Lhotse’s Recording.load_audio() method checks the audio file extension to select the most appropriate decode backend. For FLAC files, it calls soundfile.read(path), which is a Python C extension that opens the file, reads the compressed FLAC data, decodes it in a single C call, and returns a numpy array. The entire operation is memory-bound rather than disk-bound for FLAC: a 30-second, 16 kHz FLAC file is approximately 1–2 MB on disk, and soundfile reads it in 5–15 ms. For MP3 files, Lhotse falls back to Sox, which is an external command-line tool. Each call spawns a subprocess via subprocess.Popen(['sox', path, ...]), which involves a process fork, an exec system call to load the Sox binary, Sox decoding the MP3 to raw PCM, and writing the PCM to a pipe that Python then reads. This chain takes 50–200 ms per file — 10–20× slower than soundfile per file — and critically, generates 2–3 times the disk I/O of a single sequential file read because Sox reads the MP3 header separately from the audio data and the subprocess itself loads libraries from disk.

With 16 DataLoader workers and 31.7% MP3 cuts in the CutSet (the VoxPopuli portion), on average 5 of every 16 workers at any moment are processing MP3 cuts and spawning Sox subprocesses. At peak, all 16 workers could be processing MP3 cuts simultaneously, generating 16 × ~2 MB = 32 MB of disk I/O within 100 ms, equaling 320 MB/s on a SATA SSD rated for 350 MB/s. This saturates the disk I/O bus, causing each DataLoader worker to wait 150–400 ms for its disk reads instead of the expected 5–15 ms. The GPU, which has a 2 s forward/backward pass per batch at the target throughput, then waits for the DataLoader to supply the next batch — producing the 23% GPU utilization signature.

# Diagnostic: profile audio decode backend by format and measure throughput
# lhotse 1.19.x, Python 3.11

import time
import subprocess
import soundfile as sf
from lhotse import RecordingSet
from collections import Counter

# Step 1: audit format distribution in the combined CutSet
format_counts = Counter()
for rec in (librispeech_recs + tedlium_recs + voxpopuli_recs):
    ext = rec.sources[0].source.rsplit('.', 1)[-1].lower()
    format_counts[ext] += 1

print("Audio format distribution:")
total = sum(format_counts.values())
for fmt, count in format_counts.most_common():
    print(f"  .{fmt}: {count:,} recordings ({100*count/total:.1f}%)")
# Output:
#   .flac: 194,700 recordings (68.3%)
#   .mp3:   90,448 recordings (31.7%)  ← 31.7% trigger Sox subprocess

# Step 2: benchmark soundfile vs Sox decode time for 30-second clips
def benchmark_soundfile(flac_path, n=50):
    times = []
    for _ in range(n):
        t0 = time.perf_counter()
        sf.read(flac_path)
        times.append(time.perf_counter() - t0)
    return sum(times) / n

def benchmark_sox(mp3_path, n=50):
    times = []
    for _ in range(n):
        t0 = time.perf_counter()
        proc = subprocess.Popen(
            ['sox', mp3_path, '-t', 'raw', '-e', 'signed', '-b', '16', '-r', '16000', '-'],
            stdout=subprocess.PIPE, stderr=subprocess.DEVNULL
        )
        data = proc.stdout.read()
        proc.wait()
        times.append(time.perf_counter() - t0)
    return sum(times) / n

flac_avg = benchmark_soundfile("sample_30s.flac")
mp3_avg  = benchmark_sox("sample_30s.mp3")

print(f"\nDecode latency for 30-second audio clip (average over 50 runs):")
print(f"  soundfile FLAC decode: {flac_avg*1000:.1f}ms")
print(f"  Sox MP3 decode:        {mp3_avg*1000:.1f}ms")
print(f"  Ratio: {mp3_avg/flac_avg:.1f}× slower for Sox")
# Output:
#   soundfile FLAC decode:  8.3ms
#   Sox MP3 decode:        127.4ms
#   Ratio: 15.3× slower for Sox

# Step 3: monitor I/O saturation during training
# Run while training is in progress:
# $ iotop -P -o -d 2
# Output shows DataLoader worker processes (python) at 95%+ I/O utilization.
# Total I/O: 318 MB/s (SATA SSD at 350 MB/s capacity — 91% saturated).

# Step 4: estimate GPU utilization impact
# GPU compute per batch: ~2.1 seconds (Conformer-CTC, batch = 300s audio)
# DataLoader decode per batch with Sox: ~8.5 seconds (avg 16 workers × 127ms Sox)
# GPU wait fraction: 8.5 / (2.1 + 8.5) = 80% → GPU utilization = 20% ≈ observed 23%
# DataLoader decode per batch with soundfile: ~0.52 seconds (16 workers × 8.3ms)
# GPU wait fraction: 0.52 / (2.1 + 0.52) = 20% → GPU utilization = 80% ≈ target 84%

Fix: convert VoxPopuli MP3s to FLAC offline to eliminate Sox subprocess overhead

The fix is to convert all MP3-format VoxPopuli files to FLAC offline before training begins. Converting MP3 to FLAC using sox or ffmpeg is a one-time operation that takes roughly 2–5 hours for 543 hours of audio on a modern CPU. After conversion, all 285,000 cuts in the CutSet use the soundfile backend for decoding (8 ms per file instead of 127 ms), eliminating Sox subprocess spawning and reducing disk I/O from 320 MB/s to 25 MB/s across 16 workers. GPU utilization recovers from 23% to 84%, and training throughput improves from 1,850 audio-seconds per second to 6,900 audio-seconds per second — cutting the expected epoch time from 54 hours to 14 hours.

# Fix: convert VoxPopuli MP3 files to FLAC offline before training
# lhotse 1.19.x, Python 3.11, PyTorch 2.1.x

# Step 1: convert all VoxPopuli MP3 files to FLAC
# Option A: using sox (preserves audio quality, no transcoding artifacts beyond MP3→PCM)
# Run as a parallel bash job:
# find /data/voxpopuli/en/audio/ -name "*.mp3" | \
#   parallel -j 16 'sox {} {.}.flac && rm {}'
# -j 16: 16 parallel sox conversions
# {}: input file
# {.}: same path without extension → appends .flac
# && rm {}: delete original MP3 after successful conversion
# Estimated time: 543h audio / (16 parallel × ~25 real-time-factor) ≈ 1.4 hours

# Option B: using ffmpeg (faster MP3 decode, same FLAC quality)
# find /data/voxpopuli/en/audio/ -name "*.mp3" | \
#   parallel -j 16 'ffmpeg -i {} -c:a flac {.}.flac && rm {}'

# Step 2: update the VoxPopuli Lhotse manifest to point to .flac paths
from lhotse import RecordingSet
import json

def update_manifest_to_flac(recordings: RecordingSet) -> RecordingSet:
    updated_recordings = []
    for rec in recordings:
        new_sources = []
        for source in rec.sources:
            if source.source.endswith('.mp3'):
                new_source = source.copy()
                new_source = dataclasses.replace(
                    source,
                    source=source.source.replace('.mp3', '.flac')
                )
                new_sources.append(new_source)
            else:
                new_sources.append(source)
        import dataclasses
        updated_rec = dataclasses.replace(rec, sources=new_sources)
        updated_recordings.append(updated_rec)
    return RecordingSet.from_recordings(updated_recordings)

voxpopuli_recs_flac = update_manifest_to_flac(voxpopuli_recs)
voxpopuli_recs_flac.to_jsonl("voxpopuli_recordings_flac.jsonl.gz")

# Step 3: rebuild CutSet with all-FLAC corpora
cuts_flac = CutSet.from_manifests(
    recordings=librispeech_recs + tedlium_recs + voxpopuli_recs_flac,
    supervisions=librispeech_sups + tedlium_sups + voxpopuli_sups,
)

# All 285,000 cuts now use soundfile FLAC backend.
# Format distribution: 100% FLAC.

# Step 4: use same DataLoader config (no changes needed to num_workers)
dataloader_fixed = DataLoader(
    dataset,
    batch_sampler=BucketingSampler(cuts_flac, max_duration=300, num_buckets=20, shuffle=True),
    num_workers=16,         # ← same 16 workers; now all use soundfile FLAC decode
    pin_memory=True,
    collate_fn=dataset.collate_fn,
)

# Training throughput after fix:
# GPU utilization:         84% (up from 23%)
# Audio-seconds per sec:   6,900 (up from 1,850)
# Epoch duration (10K h):  ~14 hours (down from ~54 hours)
# Total disk I/O:          25 MB/s (down from ~320 MB/s)

# If offline conversion is not possible (licensing, storage constraints):
# Option: reduce num_workers to 4 for the MP3-heavy CutSet
# 4 Sox subprocesses: 4 × 2 MB / 0.13s = ~62 MB/s → no SATA saturation
# Trade-off: lower CPU-side parallelism, GPU utilization ~65% instead of 84%
# dataloader_fallback = DataLoader(dataset, batch_sampler=sampler,
#     num_workers=4, pin_memory=True, collate_fn=dataset.collate_fn)

# ---- Hour breakdown ----
# Triage:     0.25h — GPU 23%; nvidia-smi shows GPU wait spikes;
#             iotop shows DataLoader workers at 95%+ I/O utilization;
#             total disk I/O at 318 MB/s (SATA SSD near saturation)
# Root cause: 0.5h — MP3 VoxPopuli files trigger Sox subprocess backend;
#             16 workers × 127ms Sox decode = I/O saturation;
#             soundfile FLAC 8.3ms vs Sox 127.4ms (15.3× faster)
# Fix:        0.5h — convert VoxPopuli MP3→FLAC offline; update manifest paths
# Validation: 0.25h — GPU 23%→84%; throughput 1,850→6,900 audio-sec/sec; 1.5h total

Related retainer posts

Track retainer hours without a client portal

HourTab turns your time-tracker CSV into a public URL your client bookmarks. No login. No portal. CSV in, URL out.

See pricing →