CTransformers developer on retainer: AutoModelForCausalLM.generate() accumulates stateful KV cache across calls — developer also passes full conversation history in prompt string — each prior turn appears twice in context — model produces contradictory responses — 3 weeks — ctransformers developer on monthly retainer

ctransformers 0.2.x • Python 3.11 • macOS Apple M2 • Llama-2-7B GGUF q4_0 • AutoModelForCausalLM.from_pretrained() • generate() retains KV cache state between calls • full conversation history also passed in prompt string • turn 5: each prior turn appears twice in effective context • model contradicts earlier statements • repeats conversation back verbatim • developer adjusts temperature and prompt format — 3 weeks • fix: call model.reset() before each generate() • 1.5h retainer engagement

Northfield CPU Inference Ltd. built a customer support chatbot using ctransformers and a Llama-2-7B GGUF model on an Apple M2 Mac. They loaded the model with AutoModelForCausalLM.from_pretrained('TheBloke/Llama-2-7B-GGUF', model_type='llama', model_file='llama-2-7b.Q4_K_M.gguf') and implemented a conversation loop that maintained history by appending each turn to an accumulated prompt string. Single-turn inference worked correctly. But starting around turn 5, the model began producing responses that contradicted what it had said two turns earlier, repeated entire sections of the conversation back verbatim, or gave answers that seemed to reference two separate parallel conversations simultaneously. The developer spent three weeks adjusting temperature, switching prompt templates, and reformatting the conversation structure before engaging a ctransformers retainer developer.

The conversation implementation at the time of the engagement:

# chatbot.py — Northfield CPU Inference Ltd.
# ctransformers 0.2.x, Python 3.11, macOS Apple M2

from ctransformers import AutoModelForCausalLM

llm = AutoModelForCausalLM.from_pretrained(
    'TheBloke/Llama-2-7B-GGUF',
    model_type='llama',
    model_file='llama-2-7b.Q4_K_M.gguf',
)

SYSTEM_PROMPT = "[INST] <<SYS>>\nYou are a helpful assistant for Northfield CPU Inference Ltd.\n<</SYS>>\n\n"

# Conversation state — accumulated as prompt string
# PROBLEM: ctransformers generate() retains KV cache across calls
#          Developer also passes full history in prompt → doubled context
conversation_history = SYSTEM_PROMPT

def chat(user_message):
    global conversation_history
    conversation_history += f"{user_message} [/INST] "

    # At turn 5: conversation_history contains turns 1-4 as text
    # KV cache also contains turns 1-4 from prior generate() calls
    # model sees turns 1-4 TWICE: once in KV cache, once in prompt string
    # → double-weighted attention on prior context
    # → model generates responses merging two copies of the conversation

    response = llm(conversation_history, max_new_tokens=256)
    conversation_history += f"{response} [INST] "
    return response

# Turns 1-4: subtle repetition; usually dismissed as model quirks
# Turn 5: model explicitly repeats user's earlier question verbatim
#          then answers the repeated version, not the current question
# Turn 6: model contradicts a fact it stated in turn 2
#          (both copies of turn 2 attend at ~2x weight → unstable generation)

Why ctransformers generate() is stateful by default

Unlike llama-cpp-python’s Llama() class, which starts each call with a fresh context by default, ctransformers AutoModelForCausalLM.generate() is stateful. After each call, the model object retains the KV cache entries from the tokens it just processed. The next call to generate() begins with that accumulated state already in the context window. This is intentional: it allows efficient multi-turn chat by extending the KV cache one turn at a time rather than reprocessing the entire history from scratch on each call.

The problem emerges when a developer writes a ctransformers application using the mental model from other inference libraries — expecting generate() to be stateless — and simultaneously manages conversation history as an explicit accumulated string. Each call to generate() adds the current prompt to an already-populated KV cache that contains all prior turns, and the prompt string also contains all prior turns. The model’s effective context becomes: KV cache(turn 1, turn 2, turn 3, turn 4) + prompt string(turn 1, turn 2, turn 3, turn 4, turn 5).

This doubled context produces characteristic degradation. Attention weights for each prior turn are roughly doubled, pulling the model toward the content of earlier turns regardless of the current prompt. The generation attends to two simultaneous copies of the conversation context as if two parallel conversations are being superimposed. At turn 5 or 6 the degradation becomes visible: the model repeats prior answers verbatim, generates contradictions as the two attention patterns compete, or produces output that merges phrasing from early turns into responses that should reference the latest message only.

# Diagnosing the doubled-context issue
# Log model._model.n_tokens before and after each generate() call

from ctransformers import AutoModelForCausalLM

llm = AutoModelForCausalLM.from_pretrained(
    'TheBloke/Llama-2-7B-GGUF',
    model_type='llama',
    model_file='llama-2-7b.Q4_K_M.gguf',
)

SYSTEM_PROMPT = "[INST] <<SYS>>\nYou are a helpful assistant.\n<</SYS>>\n\n"
conversation_history = SYSTEM_PROMPT

messages = [
    "What is your return policy?",
    "How do I initiate a return?",
    "Is there a restocking fee?",
    "What if the item is damaged?",
    "Can I return digital purchases?",
]

for i, user_msg in enumerate(messages, 1):
    conversation_history += f"{user_msg} [/INST] "

    tokens_before = llm._model.n_tokens
    response = llm(conversation_history, max_new_tokens=64)
    tokens_after = llm._model.n_tokens
    conversation_history += f"{response} [INST] "

    print(f"Turn {i}: n_tokens before={tokens_before}, after={tokens_after}")
    print(f"  Prompt chars: {len(conversation_history)}")

# Output:
# Turn 1: n_tokens before=0,   after=178   (fresh start: only turn 1 in KV cache)
# Turn 2: n_tokens before=178, after=421   (KV cache grows: turns 1+2)
# Turn 3: n_tokens before=421, after=673   (KV cache: turns 1+2+3)
# Turn 4: n_tokens before=673, after=934   (KV cache: turns 1+2+3+4)
# Turn 5: n_tokens before=934, after=1,271 (KV cache: turns 1-4; prompt also has turns 1-4)
#          Effective context: ~1,868 tokens (934 KV + 934 prompt tokens overlapping)
#          Each prior turn attended at ~2x weight in self-attention
#          Turn 5 model output: "As I mentioned regarding your return policy question..."
#          [Repeating turn 1 answer instead of answering turn 5 about digital purchases]

The fix: call model.reset() before each generate() for stateless conversation

The fix resolves the doubling by choosing one conversation management strategy and using it exclusively. For the existing application — which already manages conversation history as an explicit accumulated string — the correct approach is to clear the KV cache before each generate() call with model.reset(). This makes generate() stateless: every call processes the full conversation history from scratch using only the prompt string, with no prior KV cache state. The behavior then matches what the developer originally expected.

# chatbot.py — FIXED (Northfield CPU Inference Ltd.)
# ctransformers 0.2.x, Python 3.11, macOS Apple M2

from ctransformers import AutoModelForCausalLM

llm = AutoModelForCausalLM.from_pretrained(
    'TheBloke/Llama-2-7B-GGUF',
    model_type='llama',
    model_file='llama-2-7b.Q4_K_M.gguf',
)

SYSTEM_PROMPT = "[INST] <<SYS>>\nYou are a helpful assistant for Northfield CPU Inference Ltd.\n<</SYS>>\n\n"

conversation_history = SYSTEM_PROMPT
turn_count = 0


def chat(user_message):
    global conversation_history, turn_count
    turn_count += 1
    conversation_history += f"{user_message} [/INST] "

    # FIXED: clear KV cache before each call
    # model.reset() sets model._model.n_tokens = 0
    # generate() processes only the prompt string — no prior KV state
    tokens_before_reset = llm._model.n_tokens
    llm.reset()
    assert llm._model.n_tokens == 0, "KV cache reset failed"

    response = llm(conversation_history, max_new_tokens=256)
    conversation_history += f"{response} [INST] "

    print(f"[Turn {turn_count}] KV before reset: {tokens_before_reset}, "
          f"after generate: {llm._model.n_tokens}, "
          f"history chars: {len(conversation_history)}")
    return response


# Validation — turn 5:
# KV before reset: 934 (accumulated from prior calls)
# After reset + generate: fresh context, no doubling
# Output at turn 5: "Digital purchases are generally non-refundable..."
#                   [Correctly answers the current question]
# No repetition, no contradiction, no doubled attention

Validation

# Validation log — November 2026
# ctransformers 0.2.x, Llama-2-7B GGUF q4_0, macOS Apple M2

# ---- Before fix ----
# Strategy: accumulated KV cache + explicit conversation history in prompt
# Turn 1: KV=0→178 — correct response
# Turn 2: KV=178→421 — slightly repetitive phrasing (dismissed)
# Turn 3: KV=421→673 — correct but oddly verbose
# Turn 4: KV=673→934 — correct; slight echo of turn 2 phrasing
# Turn 5: KV=934→1,271 — model repeats turn 1 answer verbatim as part of turn 5 response
# Turn 6: KV=1,271→1,580 — model contradicts the delivery window stated in turn 2
#          ("...returns take 3-5 days" vs earlier "...returns take 7-10 days")

# ---- After fix: model.reset() before each generate() ----
# Turn 1: KV reset 0→0, fresh generate → 178 KV
# Turn 2: KV reset 178→0, fresh generate → 421 KV
# Turn 3: KV reset 421→0, fresh generate → 673 KV
# Turn 5: KV reset 934→0, fresh generate → 1,148 KV
#          Response: directly answers turn 5 question; no repetition of prior answers

# Contradictory responses: 0/20 turns (was 6/20)             ✓
# Verbatim repetition of prior answers: 0/20 (was 4/20)      ✓
# Correct answer to current question: 20/20 (was 14/20)      ✓
# model._model.n_tokens == 0 before each call: verified       ✓

# ---- Hour breakdown ----
# Triage:     0.5h — contradictory responses at turn 5+; add n_tokens logging;
#             confirm KV cache grows across calls without reset; rule out prompt format
# Root cause: 0.25h — doubled context confirmed: n_tokens at call = prior KV + new prompt tokens
# Fix:        0.5h — add model.reset() before each generate(); add assert n_tokens==0
# Validation: 0.25h — 20-turn test; no contradictions; no repetition; answers on-topic
# Total:      1.5h

# Retainer engagement cost: 1.5h (estimated $135–210 at mid-level rates)

Related retainer posts

Track retainer hours without a client portal

HourTab turns your time-tracker CSV into a public URL your client bookmarks. No login. No portal. CSV in, URL out.

See pricing →