FlashInfer developer on retainer: paged KV cache allocated for max_seq_len÷page_size pages total — batch_size=8 each sequence needs max_seq_len÷page_size pages — page indices 128–1023 out of bounds — garbage output for sequences 1–7 — no CUDA error — 3 weeks — FlashInfer developer on monthly retainer

flashinfer 0.1.x • Python 3.11 • NVIDIA A100 80 GB • Llama-3-8B • BatchPrefillWithPagedKVCacheWrapper • kv_cache.shape[0]=128 (max_seq_len=2048÷page_size=16) • batch_size=8 sequences • paged_kv_indices for sequences 1–7 reference slots 128–1023 • CUDA reads adjacent memory silently • sequence 0 correct; sequences 1–7 nonsensical • developer checks attention mask, page_size, flash attention version — 3 weeks • fix: allocate kv_cache as max_pages_per_seq × max_batch_size pages • all 8 sequences correct • 1.5h retainer engagement

Northfield FlashInfer Research Ltd. built a custom paged-attention LLM inference server for Llama-3-8B using FlashInfer 0.1.x on an NVIDIA A100 80 GB. The developer implemented BatchPrefillWithPagedKVCacheWrapper for prefill and BatchDecodeWithPagedKVCacheWrapper for autoregressive decoding. Single-sequence inference (batch_size=1) produced correct outputs matching a reference FlashAttention-2 implementation. When the developer scaled to batch_size=8 for production, the first sequence in each batch produced correct output, but sequences 1 through 7 produced responses that were incoherent and unrelated to their prompts. No CUDA errors were raised. No NaN or Inf values appeared in the output logits. The developer spent three weeks adjusting attention masks, changing page_size from 16 to 8 and 32, upgrading FlashInfer versions, and comparing batch construction code against reference examples. The first sequence remained correct; sequences 1–7 remained wrong. The developer engaged a FlashInfer retainer developer after three weeks of debugging had not identified the cause.

The paged attention infrastructure at the time of the engagement:

# paged_attention.py — Northfield FlashInfer Research Ltd.
# flashinfer 0.1.x, Python 3.11, NVIDIA A100 80GB
# Llama-3-8B: 32 heads, head_dim=128, 32 KV heads, 32 layers
# page_size=16, max_seq_len=2048, max_batch_size=8

import torch
import flashinfer

PAGE_SIZE = 16
MAX_SEQ_LEN = 2048
MAX_BATCH_SIZE = 8
NUM_HEADS = 32
NUM_KV_HEADS = 32
HEAD_DIM = 128

MAX_PAGES_PER_SEQ = MAX_SEQ_LEN // PAGE_SIZE   # = 128

# BUG: KV cache allocated for single-sequence capacity
# shape[0] = MAX_PAGES_PER_SEQ = 128 total pages
# Enough for ONE sequence of up to MAX_SEQ_LEN tokens
# NOT enough for 8 sequences of up to MAX_SEQ_LEN tokens (needs 1024 pages)
kv_cache = torch.zeros(
    [MAX_PAGES_PER_SEQ, 2, PAGE_SIZE, NUM_KV_HEADS, HEAD_DIM],
    # ↑ 128 total pages ← only covers sequence 0's pages (indices 0-127)
    # sequences 1-7 will reference pages 128-1023 → out of bounds
    dtype=torch.float16,
    device="cuda",
)
# kv_cache.shape = [128, 2, 16, 32, 128]

# Page table allocator — assigns page indices sequentially per sequence
def build_page_table(seq_lens, max_pages_per_seq):
    """Build paged_kv_indptr and paged_kv_indices for a batch of sequences."""
    batch_size = len(seq_lens)
    paged_kv_indptr = [0]
    paged_kv_indices = []
    paged_kv_last_page_len = []

    page_counter = 0  # global page counter — starts at 0
    for seq_len in seq_lens:
        num_pages = (seq_len + PAGE_SIZE - 1) // PAGE_SIZE
        seq_page_indices = list(range(page_counter, page_counter + num_pages))
        paged_kv_indices.extend(seq_page_indices)
        paged_kv_indptr.append(paged_kv_indptr[-1] + num_pages)
        paged_kv_last_page_len.append(seq_len % PAGE_SIZE or PAGE_SIZE)
        page_counter += num_pages   # ← next sequence's pages start here

    # For batch_size=8, each seq_len=2048:
    # sequence 0: pages 0-127   (kv_cache[0..127]  — valid   ✓)
    # sequence 1: pages 128-255 (kv_cache[128..255] — OUT OF BOUNDS ✗)
    # sequence 2: pages 256-383 (kv_cache[256..383] — OUT OF BOUNDS ✗)
    # ...
    # sequence 7: pages 896-1023 (kv_cache[896..1023] — OUT OF BOUNDS ✗)
    return (
        torch.tensor(paged_kv_indptr, dtype=torch.int32, device="cuda"),
        torch.tensor(paged_kv_indices, dtype=torch.int32, device="cuda"),
        torch.tensor(paged_kv_last_page_len, dtype=torch.int32, device="cuda"),
    )

# CUDA behavior for out-of-bounds KV page reads:
# flashinfer uses kv_cache[page_idx] to fetch K and V slices per page
# page_idx=128 on a tensor of shape [128, ...] is beyond the allocation
# CUDA does NOT raise an error — reads from adjacent GPU memory addresses
# Adjacent memory may contain:
#   - zeros (from torch.zeros allocation of another tensor)
#   - activation values from previous layers
#   - weight tensors from the model
# The attention softmax over garbage K/V produces garbage attention output
# Output is not NaN/Inf — it is in a valid float16 range
# The model decoder produces text that appears fluent but is factually wrong
# Single-sequence batch (sequence 0 only): pages 0-127 — always valid ✓
# Multi-sequence batch: sequence 0 correct, sequences 1-7 nonsensical ✗

Why CUDA does not raise an error on out-of-bounds page reads

FlashInfer's paged attention CUDA kernels use paged_kv_indices as global page slot indices into the kv_cache tensor. The kernel computes the memory address of each key and value slice as kv_cache_ptr + page_idx * page_stride. When page_idx=128 and kv_cache was allocated with 128 entries (indices 0–127), the computed address points to memory 128 pages past the start of the KV cache buffer.

CUDA does not perform bounds-checking on tensor indexing within custom kernels. The GPU memory allocator (via torch.zeros) allocates a contiguous region for 128 pages. The 129th page address and beyond fall within the same CUDA virtual address space, potentially occupied by other tensors allocated before or after the KV cache. Reading from these addresses returns whatever values happen to reside there—typically zeros, intermediate activations, or weight values from other operations. The read completes without a segmentation fault because CUDA's virtual memory is flat and the addresses are within the mapped GPU memory range.

The attention computation with garbage keys and values produces attention weights that are mathematically valid (they sum to 1.0 after softmax) but meaningless. The attended values produce a context vector that contributes nonsensical information to the token prediction. Because the garbage values are in a reasonable float16 range, the output logits do not contain NaN or Inf, and the generated tokens appear syntactically fluent while being semantically wrong.

# Demonstrating the page index out-of-bounds diagnostic

import torch
import flashinfer

PAGE_SIZE = 16
MAX_SEQ_LEN = 2048
NUM_HEADS = 32
NUM_KV_HEADS = 32
HEAD_DIM = 128

# Diagnostic: check if paged_kv_indices exceed kv_cache capacity
def validate_page_table(paged_kv_indices, kv_cache):
    max_index = paged_kv_indices.max().item()
    num_pages = kv_cache.shape[0]
    if max_index >= num_pages:
        raise ValueError(
            f"paged_kv_indices.max()={max_index} >= kv_cache.shape[0]={num_pages}. "
            f"KV cache has {num_pages} pages but indices reference up to page {max_index}. "
            f"Allocate kv_cache with at least {max_index + 1} pages: "
            f"max_pages_per_seq × max_batch_size = {MAX_SEQ_LEN // PAGE_SIZE} × batch_size"
        )

# Developer's allocation (buggy — 128 pages for 8 sequences):
kv_cache_buggy = torch.zeros(
    [MAX_SEQ_LEN // PAGE_SIZE, 2, PAGE_SIZE, NUM_KV_HEADS, HEAD_DIM],
    dtype=torch.float16, device="cuda"
)
# kv_cache_buggy.shape[0] = 128

# Simulate batch of 8 sequences, each using 100 pages (1600-token prompts):
batch_size = 8
pages_per_seq = 100
paged_kv_indices = torch.arange(
    batch_size * pages_per_seq, dtype=torch.int32, device="cuda"
)
# paged_kv_indices: [0, 1, ..., 799]
# paged_kv_indices.max() = 799 >> kv_cache_buggy.shape[0] = 128

try:
    validate_page_table(paged_kv_indices, kv_cache_buggy)
except ValueError as e:
    print(e)
# ValueError: paged_kv_indices.max()=799 >= kv_cache.shape[0]=128.
# KV cache has 128 pages but indices reference up to page 799.
# Allocate kv_cache with at least 800 pages:
# max_pages_per_seq × max_batch_size = 128 × 8

The fix: allocate KV cache as max_pages_per_seq × max_batch_size pages

The fix changes the KV cache allocation from single-sequence capacity (max_seq_len // page_size) to full batch capacity ((max_seq_len // page_size) * max_batch_size). The page table construction logic does not need to change—page indices are assigned sequentially across sequences, which is correct. The only change is that the KV cache tensor has enough capacity to hold all assigned pages.

# paged_attention.py — FIXED (Northfield FlashInfer Research Ltd.)
# flashinfer 0.1.x, Python 3.11, NVIDIA A100 80GB
# Llama-3-8B: 32 heads, head_dim=128, 32 KV heads, 32 layers

import torch
import flashinfer

PAGE_SIZE = 16
MAX_SEQ_LEN = 2048
MAX_BATCH_SIZE = 8
NUM_HEADS = 32
NUM_KV_HEADS = 32
HEAD_DIM = 128

MAX_PAGES_PER_SEQ = MAX_SEQ_LEN // PAGE_SIZE   # = 128
TOTAL_PAGES = MAX_PAGES_PER_SEQ * MAX_BATCH_SIZE  # = 1024  ← FIXED

# FIXED: KV cache allocated for full batch capacity
# shape[0] = TOTAL_PAGES = 1024 total pages
# Covers batch_size=8, each sequence up to MAX_SEQ_LEN=2048 tokens
kv_cache = torch.zeros(
    [TOTAL_PAGES, 2, PAGE_SIZE, NUM_KV_HEADS, HEAD_DIM],
    # ↑ 1024 total pages ← covers all 8 sequences × 128 pages each
    dtype=torch.float16,
    device="cuda",
)
# kv_cache.shape = [1024, 2, 16, 32, 128]
# VRAM: 1024 × 2 × 16 × 32 × 128 × 2 bytes = 268MB per layer
# 32 layers: ~8.6GB total KV cache for max_batch=8, max_seq=2048

# Safety assertion — add before every batch to catch future misconfigurations:
def build_page_table(seq_lens, max_pages_per_seq, kv_cache):
    batch_size = len(seq_lens)
    paged_kv_indptr = [0]
    paged_kv_indices = []
    paged_kv_last_page_len = []

    page_counter = 0
    for seq_len in seq_lens:
        num_pages = (seq_len + PAGE_SIZE - 1) // PAGE_SIZE
        seq_page_indices = list(range(page_counter, page_counter + num_pages))
        paged_kv_indices.extend(seq_page_indices)
        paged_kv_indptr.append(paged_kv_indptr[-1] + num_pages)
        paged_kv_last_page_len.append(seq_len % PAGE_SIZE or PAGE_SIZE)
        page_counter += num_pages

    paged_kv_indices_tensor = torch.tensor(
        paged_kv_indices, dtype=torch.int32, device="cuda"
    )

    # Safety assertion: catches KV cache under-allocation
    assert paged_kv_indices_tensor.max().item() < kv_cache.shape[0], (
        f"paged_kv_indices.max()={paged_kv_indices_tensor.max().item()} "
        f">= kv_cache.shape[0]={kv_cache.shape[0]}. "
        f"Increase KV cache: max_pages_per_seq × batch_size"
    )

    return (
        torch.tensor(paged_kv_indptr, dtype=torch.int32, device="cuda"),
        paged_kv_indices_tensor,
        torch.tensor(paged_kv_last_page_len, dtype=torch.int32, device="cuda"),
    )

# After fix: for batch_size=8, each seq_len=2048:
# sequence 0: pages 0-127   (kv_cache[0..127]    — valid ✓)
# sequence 1: pages 128-255 (kv_cache[128..255]  — valid ✓)
# sequence 2: pages 256-383 (kv_cache[256..383]  — valid ✓)
# ...
# sequence 7: pages 896-1023 (kv_cache[896..1023] — valid ✓)

Validation

# Validation log — November 2026
# flashinfer 0.1.x, Llama-3-8B, NVIDIA A100 80GB
# page_size=16, max_seq_len=2048, max_batch_size=8

# ---- Before fix (kv_cache.shape[0]=128, batch_size=8) ----
# Sequence 0 output:    correct (matches reference implementation)       ✓
# Sequence 1 output:    nonsensical (unrelated to prompt)                ✗
# Sequence 2 output:    nonsensical                                      ✗
# ...
# Sequence 7 output:    nonsensical                                      ✗
# CUDA errors:          none
# Output logits NaN:    no (valid float16 range)
# Output logits Inf:    no

# ---- After fix (kv_cache.shape[0]=1024, batch_size=8) ----
# Sequence 0 output:    correct                                          ✓
# Sequence 1 output:    correct                                          ✓
# ...
# Sequence 7 output:    correct                                          ✓
# paged_kv_indices.max():  1023  < kv_cache.shape[0]=1024              ✓
# KV cache VRAM per layer: 268 MB (8.6 GB for 32 layers, float16)       ✓

# Output correctness test (10 prompt pairs, batch_size=8):
# Reference: single-sequence flashinfer inference for each prompt
# Batch inference after fix: all 8 sequences match reference logits      ✓
# Max logit delta (batch vs single): 0.001 (floating-point rounding)    ✓

# KV cache size formula for different configurations:
#   max_pages_per_seq = ceil(max_seq_len / page_size)
#   total_pages = max_pages_per_seq × max_batch_size
#
#   max_seq_len=2048, page_size=16, batch_size=8:
#     total_pages = 128 × 8 = 1024
#     VRAM per layer (float16): 1024 × 2 × 16 × 32 × 128 × 2B = 268 MB
#
#   max_seq_len=4096, page_size=16, batch_size=16:
#     total_pages = 256 × 16 = 4096
#     VRAM per layer (float16): 4096 × 2 × 16 × 32 × 128 × 2B = 1,073 MB

# ---- Hour breakdown ----
# Triage:     0.5h — single-seq correct; batch=8 sequences 1-7 wrong; no CUDA errors;
#             paged_kv_indices.max()=1023 vs kv_cache.shape[0]=128 identified;
#             OOB read mechanism confirmed: CUDA reads adjacent memory silently
# Root cause: 0.25h — KV cache shape[0]=128 covers 1 sequence max; batch needs 1024;
#             page table assigns global page indices starting at 128 for sequence 1
# Fix:        0.5h — kv_cache reallocated to shape[0]=1024; assertion added
# Validation: 0.25h — all 8 batch outputs match single-sequence reference;
#             paged_kv_indices.max()=1023 < 1024; VRAM budget confirmed
# Total:      1.5h

# Retainer engagement cost: 1.5h (estimated $143–225 at mid-level rates)

Related retainer posts

Track retainer hours without a client portal

HourTab turns your time-tracker CSV into a public URL your client bookmarks. No login. No portal. CSV in, URL out.

See pricing →