faster-whisper developer on retainer: WhisperModel("large-v2", device="cuda") defaults to device_index=0 on 4-socket A30 server — CTranslate2 does not distribute inference across multiple GPUs automatically — GPUs 1, 2, 3 idle at 0% — throughput 3.1× vs expected 12× real-time — 3 weeks — faster-whisper developer on monthly retainer

faster-whisper 0.10.x • Python 3.11 • CTranslate2 backend • WhisperModel("large-v2", device="cuda", compute_type="float16") • 4-socket x86 server • 4× NVIDIA A30 GPUs (24 GB VRAM each) • device_index=0 default • GPU 0 utilization 94% • GPU 1–3 utilization 0% • throughput 3.1× real-time (200 audio-sec/sec) • expected 12× real-time with 4× A30 • batch_size=16 (16 × 30-sec chunks in parallel) • fix: 4 separate WhisperModel instances, one per GPU (device_index=0,1,2,3); round-robin queue via multiprocessing.Pool • throughput 3.1×→11.4× real-time • 1.5h retainer engagement

Northfield faster-whisper Trans Ltd. was building a batch transcription system for an 8-hour meeting room audio recording archive. The server had four NVIDIA A30 GPUs, each with 24 GB VRAM. The developer created a single WhisperModel instance and ran transcription with batch_size=16 to process 16 audio chunks of 30 seconds in parallel: model = WhisperModel("large-v2", device="cuda", compute_type="float16") and segments, info = model.transcribe(audio, beam_size=5, task="transcribe"). During a representative 8-hour recording job, GPU 0 showed 94% utilization throughout. GPUs 1, 2, and 3 remained at 0% utilization. Total throughput was 3.1× real-time (200 audio-seconds transcribed per wall-clock second). With four A30s, the developer expected near-linear scaling to approximately 12× real-time. After three weeks of varying beam_size from 1 to 8, batch_size from 8 to 32, and compute_type between float16 and int8, throughput remained within 5% of the single-GPU baseline in all configurations. The team engaged a faster-whisper retainer developer. The root cause was that CTranslate2’s WhisperModel binds to a single CUDA device at instantiation time and does not automatically distribute inference across all available GPUs; the device="cuda" parameter with no explicit device_index defaults to device_index=0.

The faster-whisper pipeline configuration at the time of the engagement:

# Northfield faster-whisper Trans Ltd. — batch meeting room transcription
# faster-whisper 0.10.x, Python 3.11, CTranslate2 backend

from faster_whisper import WhisperModel
import os

# ---- Single-process, single-model instantiation (THE BUG) ----
model = WhisperModel(
    "large-v2",
    device="cuda",          # ← binds to device_index=0 by default
    compute_type="float16",
    # device_index not specified → defaults to 0
    # CTranslate2 does NOT auto-distribute across all visible CUDA devices.
    # nvidia-smi output during this job:
    #   GPU 0: 94% utilization, 22.1 GB / 24 GB VRAM used
    #   GPU 1:  0% utilization,  0.4 GB / 24 GB VRAM used  ← idle
    #   GPU 2:  0% utilization,  0.4 GB / 24 GB VRAM used  ← idle
    #   GPU 3:  0% utilization,  0.4 GB / 24 GB VRAM used  ← idle
)

# ---- Transcribe 8-hour recording with batch processing ----
audio_chunks = load_audio_chunks("meeting_8hr.wav", chunk_length_s=30)
# audio_chunks: list of 960 numpy arrays (8 hours × 120 chunks/hr)

results = []
for chunk in audio_chunks:
    segments, info = model.transcribe(
        chunk,
        beam_size=5,
        task="transcribe",
        language="en",
    )
    results.extend(list(segments))

# ---- Observed throughput ----
# Wall-clock time for 8-hour recording: ~148 minutes (8.87 wall-clock minutes per audio hour)
# Throughput: (8 × 3600) / (148 × 60) = 28,800 / 8,880 ≈ 3.24× real-time
# (200 audio-seconds processed per wall-clock second)
#
# Expected with 4 × A30 GPUs (single A30 baseline: ~3.1× real-time for large-v2):
# 4 × 3.1× = ~12.4× real-time
#
# Actual throughput vs expected: 3.1× vs 12× → 25.8% of expected utilization
#
# Developer checks beam_size (1, 2, 5, 8) → throughput range 3.0×–3.3× (beam_size matters ~±5%)
# Developer checks batch_size (8, 16, 32) → no effect (batch_size controls chunks in memory,
#   not GPU parallelism — CTranslate2 sequences chunks internally)
# Developer checks compute_type float16 vs int8 → int8: 3.5× real-time (marginal gain)
# Developer checks CUDA_VISIBLE_DEVICES → not set, all 4 GPUs visible
# 3 weeks: developer never checks CTranslate2 multi-GPU documentation.
# Root cause: single WhisperModel instance = single GPU process.

Why CTranslate2 does not automatically use multiple GPUs

CTranslate2 is designed around the principle of explicit device assignment: each model instance is loaded onto a single specified device and all inference for that model runs on that device. Unlike PyTorch’s DataParallel or Hugging Face Accelerate, CTranslate2 does not perform automatic data-parallel distribution across all visible CUDA devices. The device="cuda" argument maps to the first available CUDA device (index 0) unless device_index is explicitly specified. The device_index parameter can accept a list of GPU indices, but in that case CTranslate2 distributes tensor operations within a single model across those GPUs using tensor parallelism — this is intended for large models that exceed a single GPU’s VRAM, not for throughput scaling on models that fit comfortably in one GPU’s VRAM. For whisper-large-v2 (requires ~6 GB VRAM), which fits well within a 24 GB A30, the data-parallel approach (multiple independent model instances, one per GPU) is the correct scaling strategy.

The fundamental constraint is that a single Python process with a single WhisperModel instance cannot saturate four GPUs simultaneously, regardless of batch_size or beam_size tuning. The model’s inference loop runs sequentially on its assigned GPU. The only way to utilize all four GPUs for a throughput workload is to run four independent model instances in four separate workers (processes or threads), each pinned to a different GPU, with audio chunks distributed across all four workers. Python’s GIL means threading does not provide true CPU parallelism for the Python-level coordination code, but each worker’s GPU inference runs independently of the GIL — a multiprocessing.Pool of four workers achieves near-linear GPU scaling.

# Diagnostic: confirm single-GPU binding
# faster-whisper 0.10.x, Python 3.11, CTranslate2 backend

from faster_whisper import WhisperModel
import ctranslate2

# Check CTranslate2 device information
print(f"CTranslate2 version: {ctranslate2.__version__}")
print(f"CUDA available: {ctranslate2.cuda.is_available()}")
print(f"Number of CUDA devices: {ctranslate2.cuda.get_device_count()}")
# Output:
#   CTranslate2 version: 3.24.0
#   CUDA available: True
#   Number of CUDA devices: 4

# Load model and inspect its device assignment
model = WhisperModel("large-v2", device="cuda", compute_type="float16")
# Check where the model is loaded (not directly exposed, but verify via ctranslate2)
# The model's internal storage is on device_index=0 by default.

# Verify: load a model explicitly to device_index=1 and confirm it's separate
model_on_gpu1 = WhisperModel("large-v2", device="cuda", device_index=1, compute_type="float16")
# If this succeeds without OOM, GPU 1 has sufficient VRAM to hold its own copy.
# whisper-large-v2 weights: ~6 GB in float16.
# A30 VRAM: 24 GB → 4 models × 6 GB = 24 GB (fits with ~0 GB slack for activations).
# In practice: use float16 and batch_size=4–8 per worker to leave room for activations.
# Recommended: 2 models per 24 GB A30 GPU to leave headroom for activation memory.
# Total with 4 A30s: 4 × 2 = 8 model instances → ~8× throughput scaling.

Fix: instantiate one WhisperModel per GPU and distribute with multiprocessing

The fix is to instantiate one WhisperModel per GPU with an explicit device_index, run each model in its own process to avoid GIL contention and CTranslate2 internal state conflicts, and distribute audio chunks across all workers in a round-robin or work-stealing queue. With four independent WhisperModel instances pinned to device_index=0,1,2,3 and a multiprocessing.Pool dispatching audio chunks across the four workers, throughput improved from 3.1× to 11.4× real-time — 92% of the theoretical 12.4× linear-scaling ceiling, with the 8% gap attributable to Python multiprocessing coordination overhead and uneven chunk duration distribution.

# Fix: 4 WhisperModel instances, one per GPU, round-robin via multiprocessing
# faster-whisper 0.10.x, Python 3.11, CTranslate2 backend

from faster_whisper import WhisperModel
import multiprocessing as mp
import numpy as np

NUM_GPUS = 4
MODEL_NAME = "large-v2"
COMPUTE_TYPE = "float16"

# ---- Worker function (runs in separate process, owns its GPU) ----
def transcribe_worker(gpu_index: int, task_queue: mp.Queue, result_queue: mp.Queue):
    """
    Worker process: load WhisperModel on a specific GPU, process tasks from queue.
    Each worker owns its GPU exclusively — no CTranslate2 state sharing.
    """
    model = WhisperModel(
        MODEL_NAME,
        device="cuda",
        device_index=gpu_index,   # ← pin this model to its assigned GPU
        compute_type=COMPUTE_TYPE,
        num_workers=2,            # CTranslate2 internal CPU threads for pre/post-processing
    )

    while True:
        item = task_queue.get()
        if item is None:
            break  # poison pill to terminate worker
        chunk_index, audio_chunk = item
        segments_list = []
        try:
            segments, info = model.transcribe(
                audio_chunk,
                beam_size=5,
                task="transcribe",
                language="en",
            )
            segments_list = [(s.start, s.end, s.text) for s in segments]
        except Exception as e:
            segments_list = []

        result_queue.put((chunk_index, segments_list))

# ---- Main: distribute audio chunks across GPU workers ----
def transcribe_multi_gpu(audio_chunks: list) -> list:
    task_queue = mp.Queue(maxsize=NUM_GPUS * 4)
    result_queue = mp.Queue()

    # Start one worker process per GPU
    workers = []
    for gpu_idx in range(NUM_GPUS):
        p = mp.Process(
            target=transcribe_worker,
            args=(gpu_idx, task_queue, result_queue),
            daemon=True
        )
        p.start()
        workers.append(p)

    # Enqueue all chunks
    for idx, chunk in enumerate(audio_chunks):
        task_queue.put((idx, chunk))

    # Send poison pills to terminate workers after all tasks are done
    for _ in range(NUM_GPUS):
        task_queue.put(None)

    # Collect results in order
    results_unordered = {}
    for _ in range(len(audio_chunks)):
        chunk_index, segments = result_queue.get()
        results_unordered[chunk_index] = segments

    for p in workers:
        p.join()

    # Return segments in original chunk order
    all_segments = []
    for idx in range(len(audio_chunks)):
        all_segments.extend(results_unordered[idx])
    return all_segments

# ---- Run ----
audio_chunks = load_audio_chunks("meeting_8hr.wav", chunk_length_s=30)
all_segments = transcribe_multi_gpu(audio_chunks)

# ---- Results ----
# nvidia-smi during multi-GPU run:
#   GPU 0: 72% utilization, 7.8 GB / 24 GB VRAM used
#   GPU 1: 74% utilization, 7.8 GB / 24 GB VRAM used
#   GPU 2: 71% utilization, 7.8 GB / 24 GB VRAM used
#   GPU 3: 68% utilization, 7.8 GB / 24 GB VRAM used
#   (utilization lower per-GPU than single-GPU because coordination overhead)
#
# Throughput: 11.4× real-time (up from 3.1×)
# Wall-clock for 8-hour recording: ~42 min (down from ~148 min)
# GPU utilization efficiency: 11.4× / (4 × 3.1×) = 91.9% (8% coordination overhead)
#
# ---- Hour breakdown ----
# Triage:     0.25h — throughput 3.1× on 4-GPU server;
#             nvidia-smi confirms GPU 0 at 94%, GPU 1-3 at 0%
# Root cause: 0.5h — WhisperModel device='cuda' → device_index=0;
#             CTranslate2 single-process single-GPU; no auto multi-GPU distribution
# Fix:        0.5h — 4 WhisperModel instances explicit device_index=0,1,2,3;
#             multiprocessing.Pool round-robin queue; poison-pill shutdown
# Validation: 0.25h — all GPUs 68-74% utilization;
#             throughput 3.1×→11.4× real-time; 1.5h total

Related retainer posts

Track retainer hours without a client portal

HourTab turns your time-tracker CSV into a public URL your client bookmarks. No login. No portal. CSV in, URL out.

See pricing →