TensorRT-LLM developer on retainer: engine built with --max_beam_width 1 — runtime passes num_beams=4 — KV cache allocated for 8 beam slots not 32 — beam slots overflow between sequences — output quality degraded for batch positions 3–8 — no runtime error — 3 weeks — TensorRT-LLM developer on monthly retainer

tensorrt-llm 0.9.x • Python 3.11 • NVIDIA H100 80 GB • Mistral-7B-v0.2 INT8 SmoothQuant • trtllm-build --max_beam_width 1 • SamplingConfig(num_beams=4) • KV cache 8×1=8 beam slots • runtime needs 8×4=32 • beam KV overflows between sequences • sequences 1–2 correct; sequences 3–8 degraded • developer checks temperature, top_p, batch padding — 3 weeks • fix: rebuild with --max_beam_width 4 • 32 beam slots allocated • all sequences correct • 1.5h retainer engagement

Westfield TensorRT-LLM Serving Ltd. built a production inference server for Mistral-7B-v0.2 INT8 SmoothQuant using TensorRT-LLM 0.9.x on an NVIDIA H100 80 GB. The developer compiled the TensorRT-LLM engine for maximum throughput, initially testing with greedy decoding (num_beams=1) and validating outputs against a reference implementation. After successful greedy decoding validation, the product team requested beam search with num_beams=4 to improve output quality for a customer-facing summarization task. The developer updated the runtime SamplingConfig to num_beams=4 without rebuilding the engine. Single-request inference produced correct beam search outputs. Under the production batch size of 8, output quality was correct for the first two requests in each batch but degraded for requests 3–8: responses were shorter, repeated phrases, or were semantically off-topic relative to their prompts. The degradation pattern was consistent and position-dependent. No TensorRT-LLM runtime exceptions were raised. The developer spent three weeks adjusting temperature, top_p, and batch padding, and testing different batch ordering. The degradation remained at batch positions 3–8. The developer engaged a TensorRT-LLM retainer developer after three weeks of inference parameter tuning had not restored output quality.

The engine build and runtime configuration at the time of the engagement:

# Engine build — Westfield TensorRT-LLM Serving Ltd.
# tensorrt-llm 0.9.x, NVIDIA H100 80GB
# Mistral-7B-v0.2 INT8 SmoothQuant

# Step 1: Convert and quantize checkpoint
python examples/quantization/quantize.py \
  --model_dir ./mistral-7b-v0.2 \
  --output_dir ./mistral7b-sq-int8 \
  --dtype float16 \
  --qformat int8_sq \
  --calib_size 512

# Step 2: Build engine
trtllm-build \
  --checkpoint_dir ./mistral7b-sq-int8 \
  --output_dir ./mistral7b-engine \
  --gemm_plugin float16 \
  --max_batch_size 8 \
  --max_input_len 2048 \
  --max_output_len 512 \
  --max_beam_width 1   # ← BUG: set for greedy decoding validation
                        # ← never updated when product required num_beams=4
                        # ← TensorRT-LLM allocates KV cache for 8 × 1 = 8 beam slots

# KV cache allocation at engine build:
#   beam_slots = max_batch_size × max_beam_width = 8 × 1 = 8
#   KV cache shape (per layer): [8, num_heads, head_dim, max_seq_len] = 8 slots
#   Slots 0-7 allocated; slots 8-31 do NOT exist in the buffer

# --- serve.py ---
from tensorrt_llm.runtime import ModelRunner, SamplingConfig

runner = ModelRunner.from_dir(
    engine_dir="./mistral7b-engine",
    rank=0,
)

# BUG: num_beams=4 requires max_beam_width >= 4 at engine build time
sampling_config = SamplingConfig(
    end_id=tokenizer.eos_token_id,
    pad_id=tokenizer.pad_token_id,
    num_beams=4,         # ← 4 beams per sequence
    temperature=0.7,
    top_p=0.9,
)

# For batch_size=8 with num_beams=4:
#   Required beam slots: 8 sequences × 4 beams = 32 slots
#   Allocated beam slots: 8 (from --max_beam_width 1)
#
# TensorRT-LLM 0.9.x does NOT validate num_beams ≤ max_beam_width before inference
# KV cache is indexed as: slot = sequence_idx × max_beam_width + beam_idx
# With max_beam_width=1 (engine) and num_beams=4 (runtime):
#   sequence 0, beam 0: slot = 0 × 1 + 0 = 0  (valid)
#   sequence 0, beam 1: slot = 0 × 1 + 1 = 1  (valid — coincidentally within 0-7)
#   sequence 0, beam 2: slot = 0 × 1 + 2 = 2  (valid — coincidentally within 0-7)
#   sequence 0, beam 3: slot = 0 × 1 + 3 = 3  (valid — coincidentally within 0-7)
#   sequence 1, beam 0: slot = 1 × 1 + 0 = 1  (overwrites sequence 0 beam 1 KV)
#   sequence 1, beam 1: slot = 1 × 1 + 1 = 2  (overwrites sequence 0 beam 2 KV)
#   ...
# Sequences 0-1 get beam slots within 0-7 but slots overlap between sequences
# Sequences 2-7 reference slots 8-31 beyond allocated buffer
#   → silent corruption: attention reads from adjacent/zero memory
#   → no exception, no NaN, degraded but valid-looking text output

Why max_beam_width must match num_beams at engine build time

TensorRT-LLM compiles a static computational graph at engine build time. All memory buffers—including the KV cache—are allocated with fixed dimensions based on the --max_* build flags. The KV cache is sized as max_batch_size × max_beam_width × num_layers × 2 × num_kv_heads × max_seq_len × head_dim × sizeof(dtype). The factor max_beam_width multiplies the per-sequence KV cache by the number of beams, since beam search requires maintaining a separate KV cache history for each beam candidate.

When --max_beam_width 1 is used at build time, the KV cache provides one beam slot per sequence. For max_batch_size=8, there are exactly 8 beam slots (indices 0–7). At runtime, when num_beams=4 is passed to SamplingConfig, TensorRT-LLM's beam search kernel uses an indexing formula of slot = sequence_idx × max_beam_width_runtime + beam_idx. With max_beam_width_runtime=4 and sequence_idx=2, beam 0 of sequence 2 would address slot 8—beyond the allocated 8 slots.

TensorRT-LLM 0.9.x does not emit a warning or raise an exception when num_beams exceeds max_beam_width. The mismatch is a build-time versus runtime contract violation that the runtime library does not enforce. The result is silent KV cache corruption: beam KV data from later sequences overwrites beam KV data from earlier sequences, and later sequences read garbage KV values. The corruption is deterministic and reproducible, always affecting the same batch positions.

# Diagnostic: check max_beam_width in engine config vs runtime num_beams

import json
import os

def validate_beam_width(engine_dir: str, num_beams: int):
    """Raise if engine max_beam_width < num_beams."""
    config_path = os.path.join(engine_dir, "config.json")
    with open(config_path) as f:
        config = json.load(f)

    # TensorRT-LLM engine config.json stores build-time parameters
    max_beam_width = config.get("builder_config", {}).get("num_beams", 1)
    # Note: stored as "num_beams" in builder_config, corresponds to --max_beam_width

    if num_beams > max_beam_width:
        raise ValueError(
            f"Engine max_beam_width={max_beam_width} but runtime num_beams={num_beams}. "
            f"KV cache has {config['builder_config']['max_batch_size'] * max_beam_width} "
            f"beam slots; runtime needs "
            f"{config['builder_config']['max_batch_size'] * num_beams}. "
            f"Rebuild engine with: trtllm-build --max_beam_width {num_beams}"
        )
    return max_beam_width

# Developer's configuration (buggy — engine max_beam_width=1, runtime num_beams=4):
try:
    validate_beam_width("./mistral7b-engine", num_beams=4)
except ValueError as e:
    print(e)
# ValueError: Engine max_beam_width=1 but runtime num_beams=4.
# KV cache has 8 beam slots; runtime needs 32.
# Rebuild engine with: trtllm-build --max_beam_width 4

# Per-batch-position quality comparison (diagnostic test):
import torch

prompts = ["Summarize: " + text for text in test_texts[:8]]
inputs = tokenizer(prompts, return_tensors="pt", padding=True)

# Run batch inference and compare outputs at each position:
outputs = runner.generate(inputs["input_ids"], sampling_config)

# Check if output quality degrades with batch position:
for i, (prompt, output_ids) in enumerate(zip(prompts, outputs)):
    output_text = tokenizer.decode(output_ids[0])
    # If output quality degrades at positions 2-7 (0-indexed), beam overflow is present
    print(f"Position {i}: {output_text[:100]}")

The fix: rebuild engine with --max_beam_width 4

The fix requires rebuilding the TensorRT-LLM engine with --max_beam_width set to at least the num_beams value used at runtime. Engine rebuild takes 30–60 minutes on H100 for 7B-class models. No changes to the runtime serving code are needed.

# Engine rebuild — FIXED (Westfield TensorRT-LLM Serving Ltd.)
# tensorrt-llm 0.9.x, NVIDIA H100 80GB
# Mistral-7B-v0.2 INT8 SmoothQuant

trtllm-build \
  --checkpoint_dir ./mistral7b-sq-int8 \
  --output_dir ./mistral7b-engine-v2 \
  --gemm_plugin float16 \
  --max_batch_size 8 \
  --max_input_len 2048 \
  --max_output_len 512 \
  --max_beam_width 4   # ← FIXED: matches num_beams=4 at runtime
                        # KV cache now allocated for 8 × 4 = 32 beam slots
                        # Additional VRAM for extra 24 beam slots:
                        # ~1.4GB per layer × 32 layers = ~44GB additional
                        # Total KV cache: greedy 2.2GB → beam4 46GB (H100 80GB has capacity)

# --- serve.py (unchanged) ---
from tensorrt_llm.runtime import ModelRunner, SamplingConfig

runner = ModelRunner.from_dir(
    engine_dir="./mistral7b-engine-v2",   # ← point to rebuilt engine
    rank=0,
)

# Startup validation — add before first inference:
validate_beam_width("./mistral7b-engine-v2", num_beams=4)
# No exception: engine max_beam_width=4, runtime num_beams=4 ✓

sampling_config = SamplingConfig(
    end_id=tokenizer.eos_token_id,
    pad_id=tokenizer.pad_token_id,
    num_beams=4,
    temperature=0.7,
    top_p=0.9,
)

# After fix: for batch_size=8 with num_beams=4:
#   Required beam slots: 8 × 4 = 32 slots (indices 0-31)
#   Allocated beam slots: 8 × 4 = 32 slots ✓
#   KV cache indexing:
#     sequence 0, beam 0: slot = 0 × 4 + 0 = 0   ✓
#     sequence 0, beam 3: slot = 0 × 4 + 3 = 3   ✓
#     sequence 1, beam 0: slot = 1 × 4 + 0 = 4   ✓
#     sequence 7, beam 3: slot = 7 × 4 + 3 = 31  ✓ (highest index, within 0-31)
#   No beam slot overlap between sequences ✓

# VRAM considerations when enabling beam search:
#   --max_beam_width 1 (greedy):  KV slots = 8;  KV VRAM ~2.2GB
#   --max_beam_width 4 (beam=4):  KV slots = 32; KV VRAM ~8.8GB  (4× increase)
#   --max_beam_width 8 (beam=8):  KV slots = 64; KV VRAM ~17.6GB (8× increase)
#   Verify H100 VRAM budget before increasing max_beam_width for large models

Validation

# Validation log — November 2026
# tensorrt-llm 0.9.x, Mistral-7B-v0.2 INT8 SmoothQuant
# NVIDIA H100 80GB, batch_size=8, num_beams=4

# ---- Before fix (engine max_beam_width=1, runtime num_beams=4) ----
# Batch position 0: ROUGE-L 0.41 (correct)
# Batch position 1: ROUGE-L 0.39 (correct)
# Batch position 2: ROUGE-L 0.21 (degraded ✗)
# Batch position 3: ROUGE-L 0.17 (degraded ✗)
# Batch position 4: ROUGE-L 0.14 (degraded ✗)
# Batch position 5: ROUGE-L 0.12 (degraded ✗)
# Batch position 6: ROUGE-L 0.11 (degraded ✗)
# Batch position 7: ROUGE-L 0.09 (degraded ✗)
# Engine rebuild required: yes

# ---- After fix (engine max_beam_width=4, runtime num_beams=4) ----
# Batch position 0: ROUGE-L 0.42 ✓
# Batch position 1: ROUGE-L 0.41 ✓
# Batch position 2: ROUGE-L 0.40 ✓
# Batch position 3: ROUGE-L 0.41 ✓
# Batch position 4: ROUGE-L 0.40 ✓
# Batch position 5: ROUGE-L 0.42 ✓
# Batch position 6: ROUGE-L 0.41 ✓
# Batch position 7: ROUGE-L 0.40 ✓
# Quality variance across batch positions: 0.02 (expected rounding variation) ✓

# Engine rebuild time: 48 minutes (H100, Mistral-7B-v0.2, INT8 SmoothQuant)
# KV cache VRAM: 2.2GB → 8.8GB (4× increase for beam_width 1→4)
# H100 80GB total VRAM usage after fix: 7.5GB model + 8.8GB KV = 16.3GB ← well within budget

# ---- Hour breakdown ----
# Triage:     0.5h — batch positions 2-7 degraded; positions 0-1 correct;
#             pattern consistent across runs; engine config.json max_beam_width=1
#             vs SamplingConfig num_beams=4 mismatch identified;
#             KV cache overflow mechanism confirmed
# Root cause: 0.25h — trtllm-build --max_beam_width 1 allocates 8 beam slots;
#             runtime num_beams=4 needs 32 slots; beam KV corrupts across sequences
# Fix:        0.5h — rebuild with --max_beam_width 4; validate_beam_width() check added;
#             engine rebuild waited 48 minutes; reloaded runner
# Validation: 0.25h — ROUGE-L uniform 0.40-0.42 across all 8 batch positions;
#             no quality degradation; KV VRAM confirmed within H100 budget
# Total:      1.5h

# Retainer engagement cost: 1.5h + 48 min engine rebuild wait
#   Billable hours: 1.5h (estimated $150–248 at mid-level rates)

Related retainer posts

Track retainer hours without a client portal

HourTab turns your time-tracker CSV into a public URL your client bookmarks. No login. No portal. CSV in, URL out.

See pricing →