llamafile developer on retainer: default --ctx-size 512 silently truncates 2,000–4,000 token documents — summarization pipeline sees only first two paragraphs — summaries accurate for introduction, fabricated for remainder — developer never checks --ctx-size against document length — 3 weeks — llamafile developer on monthly retainer

llamafile 0.8.x • Python 3.11 • macOS Apple M2 • Mistral-7B-Instruct Q4_K_M GGUF • /v1/completions OpenAI-compatible endpoint • default --ctx-size 512 • documents of 2,000–3,800 tokens silently truncated at 512 • HTTP 200 returned — no error, no warning • model generates accurate summaries for introduction only • fabricates remainder • developer adjusts max_tokens and prompt format — 3 weeks • fix: launch with --ctx-size 8192 • pre-call token count assertion • 1.5h retainer engagement

Northfield Local LLM Corp. built a document summarization service using llamafile to run a Mistral-7B-Instruct model locally on an Apple M2 Mac. They downloaded the llamafile executable for Mistral-7B-Instruct Q4_K_M and launched it with ./mistral-7b-instruct.llamafile --server, using the default configuration. Their Python pipeline sent legal and financial documents ranging from 2,000 to 3,800 tokens to the /v1/completions endpoint with a summarization prompt. Short documents under 200 tokens were summarized correctly. But for their actual production documents, the summaries were consistently and specifically wrong in the same way: the first one or two paragraphs were summarized accurately, and then the model described document contents that were not present in the actual document, or produced generic conclusions that could apply to any document in that category. The developer spent three weeks reformatting the prompt, adjusting the system message, lowering the temperature, and testing different max_tokens values before engaging a llamafile retainer developer.

The pipeline code and launch configuration at the time of the engagement:

# Launch command — Northfield Local LLM Corp.
# llamafile 0.8.x, macOS Apple M2, Mistral-7B-Instruct Q4_K_M

# BUG: --ctx-size not specified; llamafile defaults to --ctx-size 512
# Documents of 2,000-3,800 tokens + 48-token system prompt = 2,048-3,848 tokens total
# All production documents exceed ctx-size=512; all silently truncated
./mistral-7b-instruct.llamafile --server --port 8080

# --- pipeline.py ---
import requests

SYSTEM_PROMPT = "Summarize the following document. Include all key findings, conclusions, and recommendations."
# 48 tokens — developer shortened this from 95 tokens hoping to fit more document content
# (didn't help: 512 - 48 = 464 document tokens available; still far less than needed)

def summarize_document(document_text: str) -> str:
    prompt = f"[INST] {SYSTEM_PROMPT}\n\n{document_text} [/INST]"

    # This call ALWAYS returns HTTP 200, even when document is silently truncated
    # There is no indication in the response that truncation occurred
    response = requests.post("http://localhost:8080/v1/completions", json={
        "prompt": prompt,
        "max_tokens": 256,
        "temperature": 0.3,   # Developer tried 0.1, 0.3, 0.5, 0.7 — no improvement
        "stop": ["[INST]"],
    })
    return response.json()["choices"][0]["text"]


# What the pipeline actually sends for a 2,400-token document:
# Total prompt tokens: 48 (system) + 2,400 (document) = 2,448 tokens
# llamafile ctx-size=512: silently truncates at token 512
# Model receives: 48 system tokens + first 464 tokens of document (~2 paragraphs of a 25-page legal brief)
# Model never sees: paragraphs 3-25, all key findings, all conclusions, all recommendations

# Failure pattern (2,400-token legal brief):
# Expected summary: "The brief addresses three contract breach claims... [accurate description of all 3]"
# Actual summary:   "The brief introduces a contract dispute between parties A and B in the context of..."
#                   [accurate for intro] "...and concludes that the claims are likely to succeed."
#                   [fabricated — model never saw the conclusions section]

Why llamafile defaults to --ctx-size 512 and truncates silently

llamafile is a single-file executable that packages a GGUF model, the llama.cpp inference engine, and a built-in HTTP server into one self-contained binary using the Cosmopolitan Libc toolchain. When launched with --server, it runs the llama.cpp HTTP server with default settings. The default context size in llama.cpp’s server mode is --ctx-size 512 in llamafile 0.8.x — a conservative default from llama.cpp designed for environments where available RAM is unknown. The default is documented but not prominently displayed during server startup; the launch log shows the model loading but does not print a visible warning that the context window is only 512 tokens.

When a request arrives with more tokens than --ctx-size, the llama.cpp server truncates the input to the context window. For a POST to /v1/completions or /v1/chat/completions, the server processes as many tokens as the context window allows, starting from the beginning of the prompt, and silently drops the remainder. The HTTP response is 200 OK with a valid JSON completion. The usage.prompt_tokens field in the response reports only the tokens that were actually processed (up to ctx-size), not the tokens that were sent. A caller who checks only the HTTP status code and the completion text will not detect that truncation occurred.

For the Northfield pipeline, the combined prompt (48-token system message plus document) reached 3,848 tokens for the longest documents. With ctx-size=512, the model processed 48 system tokens plus the first 464 tokens of the document — approximately the first two paragraphs of a 25-page legal brief. The rest of the document, including every section heading, every key finding, every conclusion, and every recommendation, was dropped before the model began generating. The model generated plausible-sounding summaries because it received a coherent introduction; it fabricated the rest because it had no information about what followed.

# Diagnosing the ctx-size truncation issue
# Check usage.prompt_tokens in the response; compare to actual input tokens

import requests
from transformers import AutoTokenizer

# Use any compatible tokenizer to count tokens before sending
tokenizer = AutoTokenizer.from_pretrained("mistralai/Mistral-7B-Instruct-v0.2")

SYSTEM_PROMPT = "Summarize the following document. Include all key findings, conclusions, and recommendations."

test_documents = {
    "short_doc": short_text,    # 180 tokens  — within ctx-size
    "medium_doc": medium_text,  # 1,200 tokens — exceeds ctx-size
    "long_doc": long_text,      # 2,800 tokens — exceeds ctx-size
}

for name, doc in test_documents.items():
    prompt = f"[INST] {SYSTEM_PROMPT}\n\n{doc} [/INST]"
    input_token_count = len(tokenizer.encode(prompt))

    response = requests.post("http://localhost:8080/v1/completions", json={
        "prompt": prompt,
        "max_tokens": 256,
        "temperature": 0.3,
    })
    data = response.json()
    reported_tokens = data["usage"]["prompt_tokens"]

    print(f"{name}: input_tokens={input_token_count}, "
          f"reported_prompt_tokens={reported_tokens}, "
          f"truncated={'YES' if reported_tokens < input_token_count else 'NO'}")

# Output:
# short_doc:  input_tokens=228,  reported_prompt_tokens=228, truncated=NO
# medium_doc: input_tokens=1248, reported_prompt_tokens=512, truncated=YES — 736 tokens dropped
# long_doc:   input_tokens=2848, reported_prompt_tokens=512, truncated=YES — 2336 tokens dropped

# long_doc: 82% of the document never reached the model
# Developer's pipeline was summarizing only the first 18% of every long document

The fix: launch llamafile with an explicit --ctx-size matching the application’s document lengths

The fix requires passing --ctx-size explicitly when launching llamafile, set to a value that accommodates the full prompt: system message tokens plus maximum document tokens plus response token headroom. For the Northfield pipeline with documents up to 3,800 tokens, a system prompt of 48 tokens, and max_tokens=256, the minimum safe --ctx-size is 3,800 + 48 + 256 + 64 buffer = 4,168 tokens. Setting --ctx-size 8192 provides comfortable headroom for all production documents. A pre-call token count assertion in the Python pipeline prevents silent truncation from occurring if a document exceeds the configured limit.

# FIXED launch command — Northfield Local LLM Corp.
# llamafile 0.8.x, macOS Apple M2, Mistral-7B-Instruct Q4_K_M

# FIXED: explicit --ctx-size sized for production document lengths
# Max document: 3,800 tokens + system prompt 48 tokens + max_tokens 256 + 64 buffer = 4,168
# Setting 8192 provides generous headroom; RAM increase ~1.5 GB on M2 (acceptable)
CTX_SIZE=8192
./mistral-7b-instruct.llamafile --server --port 8080 --ctx-size $CTX_SIZE

# --- pipeline.py FIXED ---
import requests
from transformers import AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained("mistralai/Mistral-7B-Instruct-v0.2")

CTX_SIZE = 8192
MAX_TOKENS = 256
SYSTEM_PROMPT = "Summarize the following document. Include all key findings, conclusions, and recommendations."
SYSTEM_TOKENS = len(tokenizer.encode(f"[INST] {SYSTEM_PROMPT}\n\n [/INST]"))  # ~55


def summarize_document(document_text: str) -> str:
    # Pre-call token count check
    doc_tokens = len(tokenizer.encode(document_text))
    total_tokens = SYSTEM_TOKENS + doc_tokens + MAX_TOKENS

    if total_tokens > CTX_SIZE:
        raise ValueError(
            f"Document too long for configured ctx-size: "
            f"system={SYSTEM_TOKENS} + doc={doc_tokens} + max_tokens={MAX_TOKENS} "
            f"= {total_tokens} > ctx_size={CTX_SIZE}. "
            f"Truncate document to {CTX_SIZE - SYSTEM_TOKENS - MAX_TOKENS - 64} tokens."
        )

    prompt = f"[INST] {SYSTEM_PROMPT}\n\n{document_text} [/INST]"

    response = requests.post("http://localhost:8080/v1/completions", json={
        "prompt": prompt,
        "max_tokens": MAX_TOKENS,
        "temperature": 0.3,
        "stop": ["[INST]"],
    })
    data = response.json()

    # Verify no truncation occurred
    assert data["usage"]["prompt_tokens"] == SYSTEM_TOKENS + doc_tokens, (
        f"Unexpected truncation: sent {SYSTEM_TOKENS + doc_tokens} tokens, "
        f"server reported {data['usage']['prompt_tokens']}"
    )

    return data["choices"][0]["text"]


# After fix: all documents up to 3,800 tokens summarized accurately
# No fabricated conclusions; all key findings present in summaries

Validation

# Validation log — November 2026
# llamafile 0.8.x, Mistral-7B-Instruct Q4_K_M, macOS Apple M2

# ---- Before fix (default --ctx-size 512) ----
# 200-token document:   summary accurate ✓ (fits in 512)
# 600-token document:   summary describes first ~6 paragraphs, stops mid-analysis
# 1,200-token document: summary describes introduction only (464 doc tokens received)
# 2,800-token document: summary accurate for first 2 paragraphs, fabricates the rest
# 3,800-token document: same — only first 464 tokens of document processed
# Developer tried: shorter system prompt (95→22 tokens) → gained 73 tokens → no meaningful change
#                  lower temperature → summaries still wrong
#                  different max_tokens → no effect on truncation

# ---- After fix (--ctx-size 8192) ----
# 200-token document:   summary accurate ✓
# 600-token document:   summary accurate ✓ (all sections present)
# 1,200-token document: summary accurate ✓ (all key findings present)
# 2,800-token document: summary accurate ✓ (all conclusions and recommendations present)
# 3,800-token document: summary accurate ✓ (full document processed: 3,848 tokens of 8,192 ctx)

# Test set: 50 documents, 180–3,800 tokens, manually verified summaries
# Summaries containing key findings from document body: 50/50 (was 12/50)  ✓
# Fabricated conclusions not present in document: 0/50 (was 38/50)          ✓
# usage.prompt_tokens == actual_input_tokens assertion: passes on all 50    ✓

# RAM usage increase: ~1.5 GB (8192-token ctx vs 512-token ctx on M2 with Metal)
# M2 Mac with 16 GB unified memory: 1.5 GB increase is acceptable
# Inference latency: unchanged for documents < 2,000 tokens; +8% for 3,800-token documents

# ---- Hour breakdown ----
# Triage:     0.5h — fabricated summaries on long documents; test with short doc → correct;
#             measure token counts with transformers tokenizer; compare to reported prompt_tokens;
#             discover 512 reported vs 2,848 sent → truncation confirmed
# Root cause: 0.25h — llamafile default --ctx-size=512; not set at launch; confirmed
# Fix:        0.5h — add --ctx-size 8192 to launch command; add pre-call token count assertion;
#             add post-call usage.prompt_tokens verification
# Validation: 0.25h — 50 documents 180–3,800 tokens; all summaries accurate
# Total:      1.5h

# Retainer engagement cost: 1.5h (estimated $127–202 at mid-level rates)

Related retainer posts

Track retainer hours without a client portal

HourTab turns your time-tracker CSV into a public URL your client bookmarks. No login. No portal. CSV in, URL out.

See pricing →