torch.compile developer on retainer: mode=“reduce-overhead” traces CUDA Graphs per input shape — variable sequence lengths trigger per-shape graph capture — 2,048 unique lengths → 2,048 CUDA Graphs → 24 GB GPU memory — OOM after hours in production — 10-second latency spikes on each new length — developer checks batch sizes not shape specialization — 3 weeks — torch.compile developer on monthly retainer

PyTorch 2.3.x • Python 3.11 • NVIDIA A100 80 GB • custom Transformer inference server • torch.compile(model, mode="reduce-overhead") • CUDA Graphs capture per input shape • variable seq_len 32–2,048 • 2,048 unique lengths → 2,048 graph captures • each capture ≈ 12 MB • 24 GB GPU memory after full range • 10-second stall per new seq_len • OOM after 8 hours in production • developer checks batch sizes, model memory, GC — 3 weeks • fix: dynamic=True • single graph handles all shapes • memory growth stops • 1.5h retainer engagement

Eastfield Torch Compile Ltd. built a custom Transformer question-answering inference server on PyTorch 2.3.x. After reading the PyTorch 2.0 release post, the developer applied torch.compile(model, mode="reduce-overhead") to their inference model. Initial benchmarks showed a 1.8× speedup on the development workload, where all test prompts had been padded to a fixed 512-token sequence length. The developer deployed the compiled server to production, where user requests arrive with arbitrary-length prompts ranging from 32 to 2,048 tokens, unpadded.

Within the first hour of production traffic, users reported intermittent 10-second response delays. The developer attributed the spikes to random GPU contention from other workloads on the host. GPU memory usage climbed slowly—approximately 12 MB every 10–30 seconds of live traffic. After 8 hours the server crashed with a CUDA out-of-memory error. The developer restarted the server, reduced batch size, and monitored more carefully. The OOM error occurred after the same duration regardless of batch size changes. GPU memory growth continued at the same rate. The developer checked for Python memory leaks, examined model memory consumption, and inspected the CUDA allocator cache over three weeks. The growth pattern persisted. The developer engaged a torch.compile retainer developer after three weeks.

The server configuration at the time of the engagement:

# Inference server — Eastfield Torch Compile Ltd.
# PyTorch 2.3.x, Python 3.11, NVIDIA A100 80GB
# Custom Transformer (12 layers, 12 heads, hidden_size=768, intermediate_size=3072)
# Weights: ~350 MB (fp16)

import torch
from fastapi import FastAPI
from transformers import AutoTokenizer

app = FastAPI()
tokenizer = AutoTokenizer.from_pretrained("./qa-model")
model = torch.load("./qa-model.pt", map_location="cuda").half().eval()

# torch.compile applied for throughput improvement
# mode="reduce-overhead" uses CUDA Graphs internally to minimize CPU-GPU synchronization
compiled_model = torch.compile(model, mode="reduce-overhead")
# BUG: mode="reduce-overhead" with CUDA Graphs requires fixed input shapes.
# CUDA Graphs capture a complete GPU execution graph for a specific input shape.
# Each unique (batch_size, seq_len) pair triggers a new CUDA Graph capture.
# A100 80GB: model=350MB + KV_cache~500MB + activations~2GB → ~3GB baseline
# Remaining 77 GB available for CUDA Graph storage
# Each CUDA Graph: ~12 MB (replay buffer + workspace + persistent fused kernels)
# 2,048 unique seq_len values × 12 MB = 24 GB CUDA Graph storage
# 24 GB + 3 GB model = 27 GB: within A100 80GB budget
# BUT: activation memory grows with seq_len for long sequences:
# seq_len=2048: activations ~8 GB; total 27 GB + 8 GB = 35 GB (still OK)
# Real issue: developer's production traffic generates seq_len 32-2048
# After 2,048 unique seq_len values seen + heavy traffic activations: OOM

@app.post("/qa")
async def answer_question(request: QARequest):
    inputs = tokenizer(
        request.question,
        request.context,
        return_tensors="pt",
        padding=False,     # ← No padding: each request has a unique seq_len
        truncation=True,
        max_length=2048,
    ).to("cuda")

    # inputs["input_ids"].shape = (1, seq_len) where seq_len varies per request
    # First request at seq_len=100:
    #   → CUDA Graph capture triggered (11-second stall)
    #   → graph stored in GPU memory (~12 MB)
    # Second request at seq_len=100:
    #   → CUDA Graph reused (8ms, as intended by mode="reduce-overhead")
    # Request at seq_len=101:
    #   → Different shape → new CUDA Graph capture (11-second stall) (~12 MB)
    # Request at seq_len=102:
    #   → Different shape → new CUDA Graph capture (11-second stall) (~12 MB)
    # ...
    # After 2,048 unique seq_len values:
    #   2,048 CUDA Graphs × 12 MB/graph = 24 GB GPU memory consumed for graphs alone
    #   10-second stall occurred 2,048 times over the 8-hour production run

    with torch.no_grad():
        outputs = compiled_model(**inputs)   # ← triggers graph capture on first new shape

    return {"answer": extract_answer(outputs, inputs)}

Why CUDA Graphs cause memory accumulation under variable input shapes

torch.compile(mode="reduce-overhead") uses CUDA Graphs to eliminate CPU-GPU synchronization overhead between kernel launches. When PyTorch captures a CUDA Graph for a specific input shape, it records all GPU kernel invocations, memory allocations, and tensor operations into a replay buffer stored in GPU memory. The replay buffer is shape-specific: a captured graph for seq_len=512 cannot be reused for seq_len=513. Each unique input shape requires its own capture.

For a Transformer model, the set of unique shapes in production equals the set of unique (batch_size, seq_len) pairs seen by the server. With variable-length inputs and batch_size=1, the number of unique shapes equals the number of distinct sequence lengths in the traffic—up to 2,048 for inputs of length 1–2,048. Each CUDA Graph consumes approximately 10–15 MB for the replay buffer plus workspace allocations from fused attention and feed-forward kernels. For a 12-layer Transformer with fused kernels, each graph is approximately 12 MB. The 10-second stalls observed in production were CUDA Graph captures triggered by the first request at each new sequence length.

# Diagnostic: check CUDA Graph accumulation and memory growth
# Run in production server process to audit compilation events

import torch

# Method 1: Check number of compilation events
# (requires torch._dynamo.compilation_metrics in PyTorch 2.3+)
def check_compilation_events():
    metrics = torch._dynamo.utils.compilation_metrics
    compilations = len(metrics)
    print(f"Total compilation events: {compilations}")
    # Healthy (dynamic=True or fixed-shape):  1-10 compilations total
    # Variable-shape without dynamic=True:    1 compilation per unique seq_len
    # After 8 hours production:               2,048 compilations seen ← BUG

# Method 2: Monitor GPU memory growth rate
import time

def memory_growth_monitor():
    """Alert if GPU memory grows by > 500 MB per minute."""
    prev_reserved = torch.cuda.memory_stats()["reserved_bytes.all.current"]
    while True:
        time.sleep(60)
        curr_reserved = torch.cuda.memory_stats()["reserved_bytes.all.current"]
        growth_mb = (curr_reserved - prev_reserved) / 1024**2
        print(f"GPU memory growth last 60s: {growth_mb:.0f} MB")
        # Healthy (dynamic=True): < 10 MB/min after warmup
        # CUDA Graph accumulation: 100-800 MB/min (12 MB per new unique shape)
        if growth_mb > 500:
            print("ALERT: GPU memory growing too fast — likely CUDA Graph accumulation")
        prev_reserved = curr_reserved

# Method 3: Measure per-request latency by shape
# If p99 latency shows bimodal distribution (8ms for seen shapes, 10,000ms for new shapes),
# CUDA Graph capture on new shapes is confirmed.
import statistics

latencies = {}  # seq_len → list of latencies

def timed_inference(compiled_model, inputs):
    seq_len = inputs["input_ids"].shape[1]
    start = torch.cuda.Event(enable_timing=True)
    end   = torch.cuda.Event(enable_timing=True)
    start.record()
    with torch.no_grad():
        outputs = compiled_model(**inputs)
    end.record()
    torch.cuda.synchronize()
    elapsed_ms = start.elapsed_time(end)
    latencies.setdefault(seq_len, []).append(elapsed_ms)
    # First call at seq_len=N: 10,000ms (CUDA Graph capture)
    # Subsequent calls at seq_len=N: 8ms (graph reuse)
    return outputs

# Diagnostic summary: after running timed_inference for 1 hour:
# Unique seq_len values seen: 1,432
# First-call latency (p100): 11,247ms  ← CUDA Graph capture
# Subsequent-call latency (p50): 8ms   ← graph reuse
# GPU memory: +17.1 GB (1,432 × 12 MB = 17.2 GB) ← matches

The fix: dynamic=True

PyTorch 2.1 introduced dynamic shape support for torch.compile(). With dynamic=True, the compiler generates a single graph using symbolic dimensions that handles all input shapes without per-shape specialization. The CUDA Graph is captured once during warmup for the first request shape; subsequent requests at any shape reuse the same compiled graph. GPU memory growth from graph accumulation stops.

# Fixed inference server — Eastfield Torch Compile Ltd.
# PyTorch 2.3.x, NVIDIA A100 80GB

import torch

model = torch.load("./qa-model.pt", map_location="cuda").half().eval()

# FIXED: dynamic=True generates a single graph with symbolic dimensions
# that handles all input shapes without per-shape specialization.
# PyTorch 2.1+ required; generates guards that verify shape constraints
# (e.g., seq_len >= 1 and seq_len <= 2048) rather than specializing per value.
compiled_model = torch.compile(
    model,
    mode="reduce-overhead",
    dynamic=True,       # ← FIXED: symbolic dimensions handle all seq_len values
    fullgraph=False,    # allow graph breaks for data-dependent control flow
)
# First request at seq_len=100:
#   → CUDA Graph captured once with symbolic seq_len (5-15 seconds warmup)
#   → graph stored: ~12 MB (single graph, not per-shape)
# Second request at seq_len=100:
#   → Symbolic graph reused (8ms) ✓
# Request at seq_len=101:
#   → Symbolic graph reused (8ms) — no new capture needed ✓
# Request at seq_len=1,500:
#   → Symbolic graph reused (8ms) — no new capture ✓
# After 2,048 unique seq_len values:
#   → 1 CUDA Graph (not 2,048) × 12 MB = 12 MB GPU memory for graphs ✓
#   → No 10-second stalls after initial warmup ✓

# Verification step: audit graph count before production deployment
def audit_compiled_model(compiled_model, test_inputs_by_shape):
    """
    Warm up the compiled model at several shapes and verify no per-shape explosion.
    """
    for seq_len in [32, 64, 128, 256, 512, 1024, 2048]:
        test_input = torch.randint(0, 32000, (1, seq_len), device="cuda")
        attention_mask = torch.ones(1, seq_len, device="cuda")
        with torch.no_grad():
            _ = compiled_model(input_ids=test_input, attention_mask=attention_mask)

    before_compilations = len(torch._dynamo.utils.compilation_metrics)
    # Warm up at 7 unique seq_len values:
    # dynamic=True: should produce 1-3 compilation events (symbolic, not per-shape)
    # static (broken): would produce 7 compilation events ← flag this case
    print(f"Compilation events after warmup: {before_compilations}")
    if before_compilations > 10:
        print("WARNING: Too many compilations — dynamic=True may not be effective")
    else:
        print("OK: Few compilations — dynamic shapes working correctly")

    mem_mb = torch.cuda.memory_stats()["reserved_bytes.all.current"] / 1024**2
    print(f"GPU memory after warmup: {mem_mb:.0f} MB")

# Optional: shape bucketing as an alternative to dynamic=True
# If dynamic=True produces graph breaks for the specific model, use bucketing instead:
BUCKET_SIZES = [32, 64, 128, 256, 384, 512, 768, 1024, 1536, 2048]

def bucket_seq_len(seq_len: int) -> int:
    """Pad seq_len to the nearest bucket size."""
    for b in BUCKET_SIZES:
        if seq_len <= b:
            return b
    return BUCKET_SIZES[-1]

# With bucketing: at most 10 unique shapes → at most 10 CUDA Graph captures
# 10 × 12 MB = 120 MB CUDA Graph storage (vs 24 GB without bucketing)
# Trade-off: some requests get more padding (up to 2× for unlucky sizes)

Validation

# Validation log — November 2026
# PyTorch 2.3.x, A100 80GB, custom Transformer (350 MB fp16)
# Production traffic: seq_len uniformly distributed 32-2048, batch_size=1

# ---- Before fix (mode="reduce-overhead", no dynamic=True) ----
# 2-hour load test, 50K requests, seq_len uniform 32-2048:
# Unique seq_len values after 2h: 1,983
# CUDA Graph captures:            1,983 (one per unique seq_len)
# CUDA Graph storage:             1,983 × 12 MB = 23.8 GB
# Total GPU memory at 2h:         27.3 GB (3 GB model + 23.8 GB graphs + 0.5 GB misc)
# Peak GPU memory at 2h:          34.1 GB (during A100 high-load burst)
# 10-second latency spikes:       1,983 (one per new unique seq_len)
# OOM occurred at:                ~8h (after all seq_len 32-2048 seen + activation spike)

# ---- After fix (mode="reduce-overhead", dynamic=True) ----
# 2-hour load test, 50K requests, seq_len uniform 32-2048:
# CUDA Graph captures:            2 (initial warmup graphs with symbolic dims)
# CUDA Graph storage:             2 × 12 MB = 24 MB
# Total GPU memory at 2h:         3.1 GB (stable; model + small symbolic graph)
# 10-second latency spikes:       2 (warmup only; first 2 request shapes)
# OOM occurred at:                never in 24-hour soak test ✓

# ---- Latency comparison (after warmup) ----
# Before fix (seen shapes):  p50=8ms,  p99=12ms  (fast — graph reuse)
# Before fix (new shapes):   p50=11,200ms, p99=11,800ms  (CUDA Graph capture stall)
# After fix (all shapes):    p50=9ms,  p99=14ms  ✓ (small dynamic overhead)

# ---- Throughput comparison ----
# Before fix (after 1h, 1,500 graphs captured): 108 req/s
# After fix (steady-state):                      101 req/s (5% slower due to dynamic overhead)
# Note: dynamic=True adds ~1ms overhead per request vs static shapes
# Tradeoff: 5% throughput reduction vs zero OOM risk and no 10-second spikes

# ---- Audit: compilation events ----
# dynamic=True warmup: torch._dynamo.utils.compilation_metrics: 2 events ✓

# ---- Memory growth rate (60-second window) ----
# Before fix: +11.9 MB/min (consistent with 1 new CUDA Graph per minute)
# After fix:  +0.1 MB/min (normal GC variance) ✓

# ---- Hour breakdown ----
# Triage:    0.5h — OOM after 8h; torch.cuda.memory_stats() shows reserved_bytes
#            growing 12 MB every 10-30 seconds; torch._dynamo.compilation_metrics
#            has 2,048 entries; pattern: 1 compilation per unique seq_len
# Root cause: 0.25h — mode='reduce-overhead' traces CUDA Graphs per input shape;
#             2,048 unique seq_len × 12 MB/graph = 24 GB; dynamic=True missing
# Fix:       0.25h — added dynamic=True; compiled_model rebuilt
# Validation: 0.5h — 2-hour soak test; memory stable; no OOM; 2 compilation events
# Total:     1.5h

Related retainer posts

Track retainer hours without a client portal

HourTab turns your time-tracker CSV into a public URL your client bookmarks. No login. No portal. CSV in, URL out.

See pricing →