CTranslate2 developer on retainer: inter_threads=8 for Whisper batch transcription — developer calls generate() in a for loop over 8 audio segments — 8 sequential single-audio calls not one batched call — throughput 1 audio/second instead of 8 — 3 weeks — CTranslate2 developer on monthly retainer

ctranslate2 3.24.x • Python 3.11 • Intel Xeon Gold 6338 • Whisper Large V3 INT8 quantized • inter_threads=8 • 8 audio segments per batch • for audio in batch: model.generate(audio) • 8 sequential calls • throughput 1 audio/second • developer increases inter_threads to 16, adjusts intra_threads, reduces beam_size — 3 weeks • fix: model.generate([audio for audio in batch]) single batched call • throughput 1→8 audio/second • 1.5h retainer engagement

Eastfield CTranslate2 Serving Ltd. built a Whisper Large V3 transcription service using CTranslate2 3.24.x with INT8 quantization on a 32-core Intel Xeon Gold 6338 server. The developer set inter_threads=8 to utilize 8 CPU cores in parallel and aimed for a throughput of 8 audio files per second. In practice, the service processed roughly 1 audio file per second regardless of how inter_threads was configured. The developer spent three weeks trying to improve throughput: increasing inter_threads from 8 to 16 (no improvement), adjusting intra_threads (no improvement), reducing beam_size from 5 to 3 (a 10% per-call speedup but still 1 file/second total). CPU utilization during transcription showed one core at 100% and seven cores idle—the opposite of what inter_threads=8 implied. The developer engaged a CTranslate2 retainer developer after three weeks of configuration changes had not moved the throughput needle.

The transcription pipeline at the time of the engagement:

# transcription.py — Eastfield CTranslate2 Serving Ltd.
# ctranslate2 3.24.x, Python 3.11, Intel Xeon Gold 6338 (32 cores)
# Whisper Large V3 INT8 quantized, inter_threads=8

import ctranslate2
import whisper
import time

# Model and processor setup
model = ctranslate2.models.Whisper(
    "whisper-large-v3-ct2-int8",
    device="cpu",
    inter_threads=8,   # ← intended for parallel throughput
    intra_threads=4,   # ← per-call parallelism
    compute_type="int8",
)
processor = whisper.load_model("large-v3").dims  # for feature extraction
feature_extractor = whisper.audio.log_mel_spectrogram  # simplified

def transcribe_batch(audio_files):
    """Transcribe a batch of 8 audio files."""
    features = [feature_extractor(audio) for audio in audio_files]

    # BUG: calling generate() in a for loop over individual audio segments
    # inter_threads=8 means up to 8 CONCURRENT generate() calls can run in parallel
    # A for loop submits them ONE AT A TIME, waiting for each to finish
    # CPU utilization: 1 core at 100%, 7 cores idle during each call
    results = []
    for feature in features:
        # Each call uses intra_threads=4 within the call
        # But calls are sequential — no parallelism between calls
        result = model.generate(
            feature,
            language=["en"],
            beam_size=5,
        )
        results.append(result)
    return results

# Performance measurement:
#   8 audio files × 25 seconds average = 200 seconds of audio
#   Wall time: ~8 seconds (8 sequential calls × ~1 second per 25s audio at 1x RT)
#   Throughput: 200 audio-seconds / 8 wall-seconds = 25 audio-seconds/second = 1x real-time
#   Or: 1 file/second (not the 8 files/second target)
#
# After increasing inter_threads to 16:
#   Still sequential calls — still 1 file/second (inter_threads doesn't help sequential loop)
#
# After reducing beam_size 5→3:
#   Per-call latency: 0.9 seconds (was 1.0) — 10% improvement
#   Throughput: 1.11 files/second — still not 8 files/second
#
# CPU utilization (htop during for loop):
#   Core 0: 100%
#   Cores 1-7: 0-5% (idle)
#   Cores 8-31: 0% (idle)
#   ← Only one generate() call active at a time; inter_threads unused

Why inter_threads does not help sequential generate() calls

In CTranslate2 3.24.x, inter_threads controls the maximum number of concurrent generate() calls that the model will process in parallel. It is a concurrency limit for simultaneous inference requests. When inter_threads=8, CTranslate2 can process up to 8 generate() calls at the same time, distributing each call across a subset of CPU cores.

A Python for loop is inherently sequential: each iteration starts only after the previous one completes. When the developer writes for feature in features: result = model.generate(feature), they submit one generate() call, wait for it to finish, then submit the next. At any given moment, exactly one generate() call is active. inter_threads=8 means CTranslate2 is ready to accept 8 simultaneous calls, but the for loop only ever provides one at a time. The inter_threads capacity goes entirely unused.

The correct usage is to pass all audio features as a list to a single generate() call. CTranslate2 interprets a list input as a batch and processes all items in parallel, up to inter_threads at a time. A batch of 8 items with inter_threads=8 processes all 8 concurrently, achieving 8x the throughput of a single item.

# Demonstrating the sequential vs batched throughput difference

import ctranslate2
import time

model = ctranslate2.models.Whisper(
    "whisper-large-v3-ct2-int8",
    device="cpu",
    inter_threads=8,
    compute_type="int8",
)

# Simulate 8 audio feature tensors (Whisper mel spectrogram format)
# In production: features = [log_mel_spectrogram(audio) for audio in audio_files]
features = [feature_batch[i] for i in range(8)]  # 8 × [80, 3000] mel spectrograms

# Sequential approach (developer's original code):
start = time.perf_counter()
sequential_results = []
for feature in features:
    result = model.generate(feature, language=["en"], beam_size=5)
    sequential_results.append(result)
sequential_time = time.perf_counter() - start
print(f"Sequential: {sequential_time:.2f}s for 8 audio files")
# Output: Sequential: 8.1s for 8 audio files
# Throughput: 1 file/second

# Batched approach (correct — passes full list to single generate() call):
start = time.perf_counter()
batched_results = model.generate(
    features,          # ← list of 8 feature tensors — CTranslate2 batches these
    language=["en"] * 8,
    beam_size=5,
)
batched_time = time.perf_counter() - start
print(f"Batched: {batched_time:.2f}s for 8 audio files")
# Output: Batched: 1.0s for 8 audio files  (inter_threads=8 fully utilized)
# Throughput: 8 files/second

# CPU utilization during batched call:
#   Core 0: 100% (generate call 1)
#   Core 1: 100% (generate call 2)
#   Core 2: 100% (generate call 3)
#   ...
#   Core 7: 100% (generate call 8)
#   Cores 8-31: 0% (beyond inter_threads=8)
# ← All 8 inter_threads active simultaneously

# inter_threads vs intra_threads distinction:
#   inter_threads=N: up to N concurrent generate() calls simultaneously
#                    → affects throughput (files/second)
#   intra_threads=M: M threads used WITHIN a single generate() call
#                    → affects latency (seconds per file)
#
# For throughput:  maximize inter_threads, batch your generate() calls
# For latency:     maximize intra_threads, one generate() call at a time

The fix: pass batch list to a single generate() call

The fix replaces the for loop over individual audio features with a single generate() call that receives all features as a list. CTranslate2 processes the list in parallel using all available inter_threads. The language and other generation parameters are passed as matching-length lists. No changes to the model, quantization, or threading configuration are needed.

# transcription.py — FIXED (Eastfield CTranslate2 Serving Ltd.)
# ctranslate2 3.24.x, Python 3.11, Intel Xeon Gold 6338 (32 cores)
# Whisper Large V3 INT8 quantized, inter_threads=8

import ctranslate2
import whisper
import time

model = ctranslate2.models.Whisper(
    "whisper-large-v3-ct2-int8",
    device="cpu",
    inter_threads=8,
    intra_threads=4,
    compute_type="int8",
)

def transcribe_batch(audio_files):
    """Transcribe a batch of 8 audio files — FIXED."""
    features = [feature_extractor(audio) for audio in audio_files]
    batch_size = len(features)

    # FIXED: pass full list to a single generate() call
    # CTranslate2 processes batch_size items in parallel using inter_threads
    # language parameter must be a list of length batch_size
    results = model.generate(
        features,                      # ← list of batch_size feature tensors
        language=["en"] * batch_size,  # ← list of batch_size language codes
        beam_size=5,
    )
    # results is a list of batch_size GenerationResult objects
    # results[0] corresponds to features[0], results[1] to features[1], etc.
    return results

    # Alternative for cases where audio arrives at different times
    # (async concurrent_generate() — available in CTranslate2 3.x):
    #
    # import concurrent.futures
    # with concurrent.futures.ThreadPoolExecutor(max_workers=8) as executor:
    #     futures = [
    #         executor.submit(model.generate, feature, language=["en"], beam_size=5)
    #         for feature in features
    #     ]
    #     results = [f.result() for f in futures]
    # This achieves the same parallelism as batched generate() when items
    # cannot be collected into a batch (streaming or variable-arrival scenarios)

# Performance after fix:
#   8 audio files × 25 seconds average = 200 seconds of audio
#   Wall time: ~1 second (8 parallel calls × 1 second each via inter_threads)
#   Throughput: 200 audio-seconds / 1 wall-second = 200 audio-seconds/second = 8x real-time
#   Or: 8 files/second ✓
#
# Optimal inter_threads configuration:
#   Set inter_threads to the batch size for full parallelism
#   On 32-core Xeon: inter_threads=8 with intra_threads=4 uses 32 cores total
#   (8 concurrent calls × 4 intra-call threads = 32 cores used)
#   Increase inter_threads beyond core_count / intra_threads causes context switching overhead

Validation

# Validation log — November 2026
# ctranslate2 3.24.x, Whisper Large V3 INT8 quantized
# Intel Xeon Gold 6338 (32 cores), inter_threads=8, intra_threads=4

# ---- Before fix (sequential generate() in for loop) ----
# Wall time for 8 audio files (25s avg):  8.1s
# Throughput:                             1.0 file/second (25 audio-seconds/second)
# CPU utilization (htop):                 Core 0: 100%, Cores 1-7: idle
# Word Error Rate:                        4.2% (transcription quality correct)

# ---- After fix (batched generate() with list input) ----
# Wall time for 8 audio files (25s avg):  1.0s                                    ✓
# Throughput:                             8.0 files/second (200 audio-seconds/sec) ✓
# CPU utilization (htop):                 Cores 0-7: 100% (all 8 inter_threads)   ✓
# Word Error Rate:                        4.2% (unchanged — batching does not      ✓
#                                               affect transcription quality)

# Throughput scaling test (varying batch size, inter_threads=8 fixed):
#   batch_size=1:   1.0s → 1 file/second (single call, full intra_threads)
#   batch_size=2:   1.0s → 2 files/second (2 parallel calls)
#   batch_size=4:   1.0s → 4 files/second (4 parallel calls)
#   batch_size=8:   1.0s → 8 files/second (8 parallel calls, inter_threads saturated) ✓
#   batch_size=16:  2.0s → 8 files/second (second batch waits for first)
#   Optimal batch_size = inter_threads for maximum throughput

# Transcription quality test:
#   10 audio files × 3 languages (English, Spanish, French)
#   Sequential results == Batched results for all 30 segments ✓
#   Word Error Rate: 4.2% English, 5.1% Spanish, 5.8% French (unchanged) ✓

# ---- Hour breakdown ----
# Triage:     0.5h — throughput 1 file/second vs 8 target; inter_threads=8 set;
#             CPU utilization: 1 core 100%, 7 cores idle; for loop identified;
#             inter_threads semantics confirmed (concurrent calls, not sequential)
# Root cause: 0.25h — CTranslate2 docs: generate() accepts list for batched processing;
#             for loop submits one call at a time; inter_threads capacity unused
# Fix:        0.5h — replace for loop with batched generate(features_list);
#             language parameter as list; output verified per-item
# Validation: 0.25h — throughput 8 files/second; CPU 8 cores 100%;
#             transcription quality unchanged; latency 1.0s for 8 files
# Total:      1.5h

# Retainer engagement cost: 1.5h (estimated $128–203 at mid-level rates)

Related retainer posts

Track retainer hours without a client portal

HourTab turns your time-tracker CSV into a public URL your client bookmarks. No login. No portal. CSV in, URL out.

See pricing →