ExLlamaV2 developer on retainer: ExLlamaV2Cache cache_size=2048 smaller than prompt_tokens + max_new_tokens — paged KV cache silently evicts oldest pages mid-generation — model loses initial prompt conditioning — repetitive degenerate output on long prompts — 3 weeks — exllamav2 developer on monthly retainer

exllamav2 0.1.x • Python 3.11 • CUDA 12.1 • NVIDIA RTX 4090 • Llama-3-8B EXL2 4.0 bpw • ExLlamaV2Cache(model, max_seq_len=2048) • 3,000-token prompts plus max_new_tokens=1,024 • total context 4,024 tokens exceeds cache_size • paged KV cache evicts initial prompt pages — silently, no exception • generation coherent for ~200 tokens then enters repetitive loop • developer adjusts temperature and repetition_penalty — 3 weeks • fix: cache_size = max(8192, prompt_len + max_new_tokens + 512) • 1.5h retainer engagement

Northfield ExLlama Inference Ltd. built a retrieval-augmented generation pipeline using ExLlamaV2 with a Llama-3-8B model quantized to 4.0 bits per weight. They loaded the model with ExLlamaV2Model and configured the KV cache with ExLlamaV2Cache(model, max_seq_len=2048). Their RAG pipeline retrieved document chunks and assembled prompts ranging from 1,000 to 3,000 tokens, then generated answers with max_new_tokens=1,024. Short prompts under 1,000 tokens produced correct, coherent responses. But for longer prompts in the 2,000–3,000 token range, the model generated coherent output for the first 200–300 tokens, then began repeating a sentence or phrase every few tokens, then locked into a tight loop over the same 5–10 tokens until the generation hit the token limit. The developer spent three weeks adjusting temperature from 0.8 down to 0.4, increasing repetition_penalty from 1.0 to 1.3, and trying different sampler configurations before engaging an ExLlamaV2 retainer developer.

The generation setup at the time of the engagement:

# rag_generate.py — Northfield ExLlama Inference Ltd.
# exllamav2 0.1.x, Python 3.11, CUDA 12.1, NVIDIA RTX 4090

from exllamav2 import ExLlamaV2, ExLlamaV2Config, ExLlamaV2Cache, ExLlamaV2Tokenizer
from exllamav2.generator import ExLlamaV2BaseGenerator, ExLlamaV2Sampler

config = ExLlamaV2Config()
config.model_dir = "/models/llama3-8b-exl2-4.0bpw"
config.prepare()

model = ExLlamaV2(config)
tokenizer = ExLlamaV2Tokenizer(config)

# BUG: cache_size=2048 is smaller than prompt_tokens + max_new_tokens for long RAG prompts
# RAG prompts: 1,000–3,000 tokens; max_new_tokens=1,024
# Total context needed: up to 4,024 tokens
# cache_size=2048 is only half the required capacity
cache = ExLlamaV2Cache(model, max_seq_len=2048)

model.load_autosplit(cache)

generator = ExLlamaV2BaseGenerator(model, cache, tokenizer)

settings = ExLlamaV2Sampler.Settings()
settings.temperature = 0.7     # Developer tried 0.4, 0.5, 0.7, 0.8 — no improvement
settings.top_p = 0.9
settings.repetition_penalty = 1.1  # Developer tried 1.0, 1.1, 1.2, 1.3 — no improvement


def generate_answer(prompt: str, max_new_tokens: int = 1024) -> str:
    # For 3,000-token prompts: cache_size=2048 is exceeded by token ~1,024 of generation
    # exllamav2 0.1.x: paged KV cache silently evicts oldest pages when full
    # Evicted pages: the beginning of the prompt (initial context, task instructions)
    # Model loses conditioning on the prompt's instructions and retrieved documents
    # Continues generating based only on recently cached tokens → repetitive loop
    output = generator.generate_simple(
        prompt,
        settings,
        max_new_tokens,
        seed=1234
    )
    # Returns tokens but cannot detect that cache eviction occurred
    return output


# Failure pattern:
# Short prompt (800 tokens):  "The key findings are: 1) revenue increased 12%..."  ✓ coherent
# Long prompt (2,800 tokens): "The key findings are: 1) revenue increased..."
#                             [coherent for ~200 tokens]
#                             "...the key findings are: 1) revenue increased..."
#                             [repetition starts — initial context evicted from cache]
#                             "...the the the findings findings findings..."
#                             [full degeneration — only last few cached tokens remain]

Why ExLlamaV2 paged KV cache evicts silently when undersized

ExLlamaV2 uses a paged KV cache architecture that allocates cache pages dynamically during generation. Each page holds a fixed number of tokens. As the generator processes the input prompt and produces new tokens, it allocates pages from the cache. When the total context—prompt tokens plus generated tokens so far—exceeds cache_size, ExLlamaV2 0.1.x begins evicting the oldest allocated pages to make room for new ones.

The eviction is silent. No exception is raised, no warning is logged, and the generate_simple() call returns normally. From the calling code’s perspective, generation completed successfully. The only observable symptom is in the output text: the model’s behavior shifts abruptly from coherent generation to repetition, because the KV cache pages that encoded the beginning of the prompt—the task instructions, the system prompt, the initial context from retrieved documents—have been evicted and the model is now conditioning only on the most recently cached tokens.

For the Northfield pipeline, the cache_size was 2,048 tokens. A 3,000-token input prompt fills the entire cache before a single output token is generated. ExLlamaV2 allocates pages for the first 2,048 prompt tokens, filling the cache, then must immediately evict those pages to allocate new ones for the remaining 952 prompt tokens. By the time the generator starts producing output, the cache contains only the last ~2,048 tokens of a 3,000-token prompt—the initial instructions and the beginning of the retrieved documents are already gone. Generation begins with the model conditioning on the middle and end of the prompt, not the beginning. The repetitive degeneration occurs because the model has no conditioning signal pointing toward a coherent completion; it drifts toward the most strongly represented tokens in the surviving cache pages and collapses into a loop.

# Diagnosing the cache undersizing issue
# Measure prompt token count before generation; compare to cache_size

from exllamav2 import ExLlamaV2, ExLlamaV2Config, ExLlamaV2Cache, ExLlamaV2Tokenizer
from exllamav2.generator import ExLlamaV2BaseGenerator, ExLlamaV2Sampler

config = ExLlamaV2Config()
config.model_dir = "/models/llama3-8b-exl2-4.0bpw"
config.prepare()

model = ExLlamaV2(config)
tokenizer = ExLlamaV2Tokenizer(config)
cache = ExLlamaV2Cache(model, max_seq_len=2048)  # Still undersized for diagnosis
model.load_autosplit(cache)
generator = ExLlamaV2BaseGenerator(model, cache, tokenizer)
settings = ExLlamaV2Sampler.Settings()
settings.temperature = 0.7

# Add pre-generation measurement
prompts = {
    "short_rag": short_prompt,    # 800 tokens  — within cache_size
    "medium_rag": medium_prompt,  # 1,800 tokens — within cache_size
    "long_rag": long_prompt,      # 3,000 tokens — exceeds cache_size
}

for name, prompt in prompts.items():
    prompt_ids = tokenizer.encode(prompt)
    prompt_tokens = prompt_ids.shape[-1]
    max_new_tokens = 1024
    total_needed = prompt_tokens + max_new_tokens
    cache_size = cache.max_seq_len  # 2048

    print(f"{name}: prompt={prompt_tokens}, max_new={max_new_tokens}, "
          f"total_needed={total_needed}, cache_size={cache_size}, "
          f"fits={'YES' if total_needed <= cache_size else 'NO — WILL EVICT'}")

# Output:
# short_rag:  prompt=800,   max_new=1024, total_needed=1824, cache_size=2048, fits=YES
# medium_rag: prompt=1800,  max_new=1024, total_needed=2824, cache_size=2048, fits=NO — WILL EVICT
# long_rag:   prompt=3000,  max_new=1024, total_needed=4024, cache_size=2048, fits=NO — WILL EVICT
#
# medium_rag already undersized — developer dismissed its subtle repetition as "model quirks"
# long_rag: prompt alone (3,000) exceeds cache_size (2,048) → eviction before first output token

The fix: size the cache to accommodate prompt plus max_new_tokens

The fix requires sizing ExLlamaV2Cache to hold the maximum expected context: prompt tokens plus max_new_tokens plus a safety buffer. For the Northfield pipeline, the maximum prompt length was 3,000 tokens and max_new_tokens was 1,024. Setting cache_size = max(8192, prompt_len + max_new_tokens + 512) ensures the cache is never smaller than the required context. A pre-generation assertion prevents silent eviction from occurring if the sizing logic is ever bypassed.

# rag_generate.py — FIXED (Northfield ExLlama Inference Ltd.)
# exllamav2 0.1.x, Python 3.11, CUDA 12.1, NVIDIA RTX 4090

from exllamav2 import ExLlamaV2, ExLlamaV2Config, ExLlamaV2Cache, ExLlamaV2Tokenizer
from exllamav2.generator import ExLlamaV2BaseGenerator, ExLlamaV2Sampler

config = ExLlamaV2Config()
config.model_dir = "/models/llama3-8b-exl2-4.0bpw"
config.prepare()

model = ExLlamaV2(config)
tokenizer = ExLlamaV2Tokenizer(config)

# FIXED: cache_size accounts for maximum prompt + max_new_tokens + buffer
# Max RAG prompt: 3,000 tokens; max_new_tokens: 1,024; buffer: 512
# cache_size = 4,536 — well under RTX 4090 VRAM budget
MAX_PROMPT_TOKENS = 3000
MAX_NEW_TOKENS = 1024
CACHE_BUFFER = 512
CACHE_SIZE = max(8192, MAX_PROMPT_TOKENS + MAX_NEW_TOKENS + CACHE_BUFFER)  # 8192

cache = ExLlamaV2Cache(model, max_seq_len=CACHE_SIZE)
model.load_autosplit(cache)

generator = ExLlamaV2BaseGenerator(model, cache, tokenizer)
settings = ExLlamaV2Sampler.Settings()
settings.temperature = 0.7
settings.top_p = 0.9
settings.repetition_penalty = 1.0  # Reverted to default — was masking the real issue


def generate_answer(prompt: str, max_new_tokens: int = MAX_NEW_TOKENS) -> str:
    # Pre-generation assertion: validate cache is large enough
    prompt_ids = tokenizer.encode(prompt)
    prompt_tokens = prompt_ids.shape[-1]
    total_needed = prompt_tokens + max_new_tokens

    if total_needed > cache.max_seq_len:
        raise ValueError(
            f"Cache undersized: prompt_tokens={prompt_tokens} + max_new_tokens={max_new_tokens} "
            f"= {total_needed} exceeds cache_size={cache.max_seq_len}. "
            f"Increase CACHE_SIZE or truncate the prompt to {cache.max_seq_len - max_new_tokens} tokens."
        )

    output = generator.generate_simple(prompt, settings, max_new_tokens, seed=1234)
    return output


# After fix: all prompts up to 3,000 tokens generate coherently
# Repetition_penalty restored to 1.0 — repetitive output was cache eviction, not sampling issue

Validation

# Validation log — November 2026
# exllamav2 0.1.x, Llama-3-8B EXL2 4.0 bpw, NVIDIA RTX 4090

# ---- Before fix (cache_size=2048) ----
# Prompt 800 tokens,   max_new=1024: coherent ✓  (total 1824 ≤ 2048)
# Prompt 1800 tokens,  max_new=1024: mostly coherent, subtle repetition at token 150+
# Prompt 2200 tokens,  max_new=1024: repetition starts at token 80
# Prompt 3000 tokens,  max_new=1024: degenerate at token 40; full loop by token 100
# Developer tried: temperature 0.4 → no improvement
#                  repetition_penalty 1.3 → suppressed repetition briefly, then worse
#                  Different sampler (top-k=40) → no improvement
# Root cause never identified — each sampler change masked symptom, never fixed cause

# ---- After fix (cache_size=8192) ----
# Prompt 800 tokens,   max_new=1024: coherent ✓  (total 1824, cache 8192)
# Prompt 1800 tokens,  max_new=1024: coherent ✓  (total 2824, cache 8192)
# Prompt 2200 tokens,  max_new=1024: coherent ✓  (total 3224, cache 8192)
# Prompt 3000 tokens,  max_new=1024: coherent ✓  (total 4024, cache 8192)
# Pre-generation assertion: validates cache capacity before each call ✓

# Coherent responses (50 prompts, 500–3,000 tokens): 50/50 (was 23/50)  ✓
# Repetitive degeneration detected: 0/50 (was 21/50)                     ✓
# Cache eviction mid-generation: 0 (was happening on 27/50 prompts)      ✓
# repetition_penalty restored to 1.0: output quality improved            ✓

# VRAM usage after fix: +1.2 GB (8192-token cache vs 2048-token cache)
# RTX 4090 has 24 GB VRAM — 1.2 GB increase is negligible
# Throughput unchanged (single-sequence generation, cache size only affects memory not speed)

# ---- Hour breakdown ----
# Triage:     0.5h — degenerate output on long prompts; confirm short prompts OK;
#             tokenize long prompts; measure prompt_tokens vs cache_size=2048;
#             3,000-token prompt exceeds cache alone; KV eviction confirmed
# Root cause: 0.25h — cache_size must exceed prompt_tokens + max_new_tokens; confirmed
# Fix:        0.5h — CACHE_SIZE = max(8192, MAX_PROMPT + MAX_NEW + BUFFER);
#             pre-generation assertion; revert repetition_penalty to 1.0
# Validation: 0.25h — 50 prompts 500–3,000 tokens; all coherent; assertion tested
# Total:      1.5h

# Retainer engagement cost: 1.5h (estimated $142–225 at mid-level rates)

Related retainer posts

Track retainer hours without a client portal

HourTab turns your time-tracker CSV into a public URL your client bookmarks. No login. No portal. CSV in, URL out.

See pricing →