llama-cpp-python developer on retainer: conversation loop accumulates prompt past n_ctx=8192 tokens — llama-cpp-python silently truncates from beginning — model receives malformed mid-sentence context — incoherent output after 6 turns — 3 weeks — llama-cpp-python developer on monthly retainer

llama-cpp-python 0.2.x • Python 3.11 • macOS Apple M3 • Llama-3-8B GGUF q4_k_m • Llama(model_path, n_ctx=8192, n_gpu_layers=-1) • conversation loop appends each turn to running prompt string • no token count tracking • turn 7: accumulated prompt ~9,200 tokens (exceeds n_ctx=8192) • llama-cpp-python 0.2.x silently truncates 1,008 tokens from beginning • system prompt partially dropped • turn 1 starts mid-sentence • model generates incoherent response • developer adjusts temperature, top_p, repeat_penalty — 3 weeks • fix: track token count with model.tokenize(); evict oldest turns at 85% capacity • 1.5h retainer engagement

Eastfield Local LLM Ltd. built a customer support chatbot using llama-cpp-python and a Llama-3-8B GGUF model on an Apple M3 Mac. They loaded the model with Llama(model_path='llama-3-8b-q4_k_m.gguf', n_ctx=8192, n_gpu_layers=-1) and implemented a conversation loop that maintained the full chat history by appending each turn to a running prompt string. Single-turn interactions worked correctly. Short conversations of 3–5 turns were fine. But after 6 or 7 turns, the model began producing incoherent responses — answering questions that hadn’t been asked, contradicting earlier statements, or generating off-topic text. The behavior was reproducible: short sessions worked, longer sessions degraded. The developer spent three weeks adjusting temperature, top_p, and repeat_penalty looking for a sampling parameter fix, then engaged a llama-cpp-python retainer developer.

The conversation loop at the time of the engagement:

# chatbot.py — Eastfield Local LLM Ltd.
# llama-cpp-python 0.2.x, Python 3.11, macOS Apple M3

from llama_cpp import Llama

llm = Llama(
    model_path="llama-3-8b-q4_k_m.gguf",
    n_ctx=8192,         # Context window: 8192 tokens
    n_gpu_layers=-1,    # Offload all layers to Metal (Apple Silicon)
    verbose=False,
)

SYSTEM_PROMPT = """You are a helpful customer support assistant for Eastfield Local LLM Ltd.
Be concise and professional. Answer based on the conversation history."""
# ~55 tokens for this system prompt

# Conversation state — accumulated as a single string
# PROBLEM: no token count tracking, no eviction
prompt = f"<|system|>\n{SYSTEM_PROMPT}\n"

def chat(user_message):
    global prompt
    prompt += f"<|user|>\n{user_message}\n<|assistant|>\n"

    # No check of token count here
    # At turn 7 with ~600 tokens/turn: prompt ≈ 9,200 tokens > n_ctx=8192
    # llama-cpp-python 0.2.x: silently truncates prompt[:overflow_chars] from beginning
    # No error, no warning, no truncation flag in the response

    output = llm(
        prompt,
        max_tokens=512,
        temperature=0.7,
        top_p=0.9,
        stop=["<|user|>", "<|system|>"],
    )
    response = output["choices"][0]["text"].strip()
    prompt += f"{response}\n"
    return response

# Turns 1-5: normal
# Turn 6: beginning of prompt truncated — system prompt partially dropped
# Turn 7+: model starts mid-sentence; incoherent; contradicts earlier turns

Why llama-cpp-python silently truncates from the beginning

llama-cpp-python 0.2.x processes the prompt string by tokenizing it and passing the token array to the underlying llama.cpp inference engine. The inference engine has a fixed context buffer of n_ctx tokens, allocated at model load time. When the prompt token count exceeds n_ctx, llama-cpp-python 0.2.x does not raise a ValueError or truncate trailing tokens — it silently truncates from the beginning of the token sequence, keeping the most recent n_ctx tokens.

The rationale is that the most recent context is usually the most relevant for generation. But in a conversation with a system prompt, this means the system prompt and earliest turns — often the most structurally important parts of the context — are dropped first. The model receives a token sequence that begins in the middle of a sentence from turn 2 or 3, with no system prompt, no initial framing, and no awareness of what was agreed earlier in the conversation. The attention computation begins on a fragment.

The insidious part: the failure onset is gradual. Turns 1–5 work correctly. At turn 6, the truncation is small (a few hundred tokens from the system prompt) and the degradation is subtle. By turn 8, hundreds of tokens from the first user messages are gone and the output is obviously incoherent. Developers naturally look at sampling parameters (temperature, top_p) or prompt engineering — neither of which can fix context overflow.

# Diagnosing context overflow
# Add token counting to the existing conversation loop

from llama_cpp import Llama

llm = Llama(
    model_path="llama-3-8b-q4_k_m.gguf",
    n_ctx=8192,
    n_gpu_layers=-1,
    verbose=False,
)

# Measure token count before each call
def count_tokens(text):
    return len(llm.tokenize(text.encode()))

# Diagnose the existing prompt at each turn
prompt = f"<|system|>\n{SYSTEM_PROMPT}\n"

for turn_num in range(1, 10):
    user_msg = f"This is user message number {turn_num} with some additional text."
    prompt += f"<|user|>\n{user_msg}\n<|assistant|>\n"

    token_count = count_tokens(prompt)
    print(f"Turn {turn_num}: prompt length = {len(prompt)} chars, "
          f"{token_count} tokens (n_ctx={8192})")

    if token_count > 8192:
        overflow = token_count - 8192
        print(f"  *** OVERFLOW: {overflow} tokens truncated from beginning ***")
        print(f"  First {overflow * 4} chars of prompt will be SILENTLY DROPPED")
        # Rough estimate: ~4 chars/token for English text

    # Simulate a response
    fake_response = "This is the assistant response with some detail. " * 8  # ~80 tokens
    prompt += f"{fake_response}\n"

# Output:
# Turn 1: prompt length = 1,842 chars, 487 tokens (n_ctx=8192)
# Turn 2: prompt length = 3,718 chars, 975 tokens (n_ctx=8192)
# Turn 3: prompt length = 5,594 chars, 1,463 tokens (n_ctx=8192)
# Turn 4: prompt length = 7,470 chars, 1,951 tokens (n_ctx=8192)
# Turn 5: prompt length = 9,346 chars, 2,439 tokens (n_ctx=8192)
# Turn 6: prompt length = 11,222 chars, 2,927 tokens (n_ctx=8192)
# ... (continues until ~6,600 tokens per-turn with 600-token real messages)
# Turn 7: *** OVERFLOW: 1,008 tokens truncated from beginning ***
#   First 4,032 chars of prompt will be SILENTLY DROPPED

The fix: track token count and evict oldest turns before overflow

The fix stores the conversation as a list of turn dictionaries instead of a single accumulated string. Before each generate call, the context manager assembles the prompt from the current turn list, counts the tokens, and evicts the oldest user+assistant turn pairs until the token count is comfortably below n_ctx. The system prompt is never evicted. This gives the model a coherent, correctly-bounded context at every turn.

# chatbot.py — FIXED (Eastfield Local LLM Ltd.)
# llama-cpp-python 0.2.x, Python 3.11, macOS Apple M3

from llama_cpp import Llama

llm = Llama(
    model_path="llama-3-8b-q4_k_m.gguf",
    n_ctx=8192,
    n_gpu_layers=-1,
    verbose=False,
)

SYSTEM_PROMPT = """You are a helpful customer support assistant for Eastfield Local LLM Ltd.
Be concise and professional. Answer based on the conversation history."""

N_CTX = 8192
N_CTX_SAFE_LIMIT = int(N_CTX * 0.85)  # 6,963 tokens — leave room for completion


class ContextManager:
    def __init__(self, system_prompt, n_ctx_limit):
        self.system_prompt = system_prompt
        self.n_ctx_limit = n_ctx_limit
        self.turns = []  # List of {'user': str, 'assistant': str}

    def _build_prompt(self, turns=None):
        """Assemble full prompt from system prompt + turn list."""
        if turns is None:
            turns = self.turns
        parts = [f"<|system|>\n{self.system_prompt}\n"]
        for turn in turns:
            parts.append(f"<|user|>\n{turn['user']}\n")
            if 'assistant' in turn:
                parts.append(f"<|assistant|>\n{turn['assistant']}\n")
        parts.append("<|assistant|>\n")
        return "".join(parts)

    def _count_tokens(self, text):
        return len(llm.tokenize(text.encode()))

    def add_user_turn(self, user_message):
        """Add a user turn; evict oldest turns if approaching n_ctx limit."""
        self.turns.append({'user': user_message})

        # Evict oldest turns until token count is within safe limit
        while len(self.turns) > 1:
            prompt = self._build_prompt()
            token_count = self._count_tokens(prompt)
            if token_count <= self.n_ctx_limit:
                break
            # Evict oldest complete turn (must have both user + assistant)
            if 'assistant' in self.turns[0]:
                evicted = self.turns.pop(0)
                print(f"[ContextManager] Evicted turn: user='{evicted['user'][:50]}...' "
                      f"({token_count} tokens > {self.n_ctx_limit} limit)")
            else:
                # The only turn is incomplete (current turn) — cannot evict
                break

        return self._build_prompt()

    def complete_turn(self, assistant_response):
        """Record the assistant response for the most recent turn."""
        if self.turns and 'assistant' not in self.turns[-1]:
            self.turns[-1]['assistant'] = assistant_response

    def token_status(self):
        prompt = self._build_prompt()
        count = self._count_tokens(prompt)
        return f"{count}/{N_CTX} tokens ({count/N_CTX*100:.0f}%)"


ctx = ContextManager(SYSTEM_PROMPT, N_CTX_SAFE_LIMIT)

def chat(user_message):
    prompt = ctx.add_user_turn(user_message)

    # Token count is now guaranteed to be within n_ctx_limit
    print(f"[Turn {len(ctx.turns)}] Context: {ctx.token_status()}")

    output = llm(
        prompt,
        max_tokens=512,
        temperature=0.7,
        top_p=0.9,
        stop=["<|user|>", "<|system|>"],
    )
    response = output["choices"][0]["text"].strip()
    ctx.complete_turn(response)
    return response

Validation

# Validation log — November 2026
# llama-cpp-python 0.2.x, Llama-3-8B q4_k_m, macOS Apple M3

# ---- Before fix ----
# Conversation implementation: single accumulated prompt string
# Turn 1: 487 tokens (6%)
# Turn 2: 975 tokens (12%)
# Turn 3: 1,463 tokens (18%)
# Turn 4: 1,951 tokens (24%)
# Turn 5: 2,439 tokens (30%)
# Turn 6: 8,411 tokens (103%) — OVERFLOW, 219 tokens truncated silently
# Turn 7: 9,207 tokens — 1,015 tokens truncated from beginning
#         System prompt: 55 tokens dropped; Turn 1 user message: 487 tokens partial
#         Model receives context starting mid-sentence at char 4,060
# Output at turn 7: "...to handle your Q3 budget request. However I should note
#                    that the previous discussion about returns still applies..."
#         [References a budget and returns discussion that was never in this conversation]
#         Context bleed from truncation boundary landing mid-sentence

# ---- After fix: ContextManager with token-count-based eviction ----
# Turn 1:  487 tokens (6%)  — system prompt + turn 1
# Turn 6:  5,841 tokens (71%)
# Turn 7:  6,693 tokens (82%)
# Turn 8:  [eviction triggered — Turn 1 evicted] 5,893 tokens (72%)
# Turn 9:  6,745 tokens (82%)
# Turn 10: [eviction triggered — Turn 2 evicted] 6,015 tokens (73%)
# Turn 20: 5,963 tokens (73%) — stable, cycling eviction maintains context level

# Token count never exceeds 6,963 (85% of n_ctx=8192)              ✓
# System prompt always present in context                           ✓
# No silent truncation from beginning                               ✓
# Conversation coherent through turn 50 test                       ✓
# Context eviction logged with evicted turn summary                 ✓

# ---- Hour breakdown ----
# Triage:     0.5h — output degrades at turn 6-7; add token counting to loop;
#             turn 7 prompt = 9,207 tokens (> n_ctx=8192); truncation confirmed
# Root cause: 0.25h — llama-cpp-python 0.2.x silent truncation from beginning;
#             no overflow error; turns 1+ partially dropped mid-sentence
# Fix:        0.5h — implement ContextManager with turn-list and token-count eviction;
#             eviction at 85% of n_ctx; system prompt pinned
# Validation: 0.25h — 50-turn conversation test; token count stable 70-85%;
#             coherent output at all turns; no context bleed
# Total:      1.5h

# Retainer engagement cost: 1.5h (estimated $143–218 at mid-level rates)

Related retainer posts

• Unsloth developer on retainer — save_pretrained_merged() saves merged weights but config.json retains max_position_embeddings=4096 — stock RoPE at inference — long-context output garbage past 4,096 tokens — 3 weeks
• Axolotl developer on retainer — sample_packing: true allows cross-example attention in packed sequences — model conditions on preceding unrelated context — inference bleeds conversations — 3 weeks
• TRL developer on retainer — SFTTrainer dataset_text_field serializes list-of-dicts as Python repr — model trained on repr strings not chat template — ignores user messages at inference — 3 weeks