MLC-LLM developer on retainer: MLCEngine maintains session state across unrelated API calls — developer never calls engine.reset() — 100 user queries accumulate in context — prior users’ document content surfaces in responses — 3 weeks — MLC-LLM developer on monthly retainer

mlc-llm 0.1.x • Python 3.11 • MacBook Pro M2 Max 96GB • Llama-3-8B INT4 W4A16 • MLCEngine created once at startup • each engine.chat.completions.create() appends to session history • query 50: 49 prior documents in context • prior users’ content referenced in current response • developer checks model_id, max_gen_len, temperature — 3 weeks • fix: engine.reset() before each independent query • response isolation restored • 1.5h retainer engagement

Northfield MLC Inference Ltd. deployed a Llama-3-8B INT4 W4A16 model using MLC-LLM 0.1.x to power a document question-answering service. Each HTTP request supplied a user’s document and a question; the service was expected to answer only from the supplied document with no memory of prior requests. The developer created a single MLCEngine at application startup and called engine.chat.completions.create() for each incoming request, passing the document and question in the user message. For the first 20 to 30 requests the service worked correctly. After approximately 50 requests, users began receiving answers that referenced terminology, company names, and facts that appeared nowhere in their submitted document. The developer spent three weeks checking the model identifier, generation length limits, and temperature parameters before engaging an MLC-LLM retainer developer.

The document Q&A service at the time of the engagement:

# qa_service.py — Northfield MLC Inference Ltd.
# mlc-llm 0.1.x, Python 3.11, MacBook Pro M2 Max 96GB
# Llama-3-8B INT4 W4A16 compiled with mlc_llm.build

import mlc_llm
from fastapi import FastAPI

app = FastAPI()

# Engine created once at startup — single shared instance
# BUG: MLCEngine 0.1.x maintains conversation history between calls
engine = mlc_llm.MLCEngine("Llama-3-8B-Instruct-q4f16_1-MLC")


@app.post("/qa")
def answer_question(document: str, question: str) -> dict:
    prompt = f"""Answer the question using only information from the document below.
If the answer is not in the document, say "Not found in document."

Document:
{document}

Question: {question}"""

    # BUG: no engine.reset() before this call
    # MLCEngine 0.1.x: each chat.completions.create() appends to internal session
    # Request 1:  context = [document_1 + question_1]
    # Request 2:  context = [document_1, answer_1, document_2 + question_2]
    # Request 50: context = [doc_1..doc_49, ans_1..ans_49, doc_50 + question_50]
    # By request 50: total tokens = 49 × avg_2100 + current = ~104,900 tokens
    # Exceeds compiled context window → engine silently evicts oldest context
    # Remaining context includes fragments from requests 20–49
    # Model answers question_50 with references to document_23's content
    response = engine.chat.completions.create(
        messages=[{"role": "user", "content": prompt}],
        max_tokens=512,
        temperature=0.1,
    )

    return {"answer": response.choices[0].message.content}


# Observed behavior:
# Request  1–25: correct answers; "Not found in document" for out-of-scope questions
# Request 50:    answer references "indemnification clause" — not in document_50
#                (a technical specification); found in document_23 (a legal contract)
# Request 75:    answer references "Helix Dynamics Inc." — not in document_75
#                (a financial report); found in document_61 (a business agreement)
# No error raised — HTTP 200 for all requests

Why MLCEngine accumulates session history across independent calls

MLCEngine in MLC-LLM 0.1.x is designed to support multi-turn conversations. It maintains an internal conversation history that persists between calls to engine.chat.completions.create(). This is the correct behavior for a chatbot application where the user expects the model to remember what was said earlier in the conversation. For a stateless Q&A service where each request is independent, this behavior is a liability: each call appends the new query and the generated answer to the engine’s growing session context.

The contamination takes two forms. The first is direct reference: the model has document_23’s content in its context window alongside document_50’s content, and it draws on both when generating an answer. The second is indirect inference: the model integrates knowledge from 49 prior documents and generates answers that reflect that accumulated knowledge base rather than only the submitted document. Both forms violate the service’s stated behavior of answering only from the current document.

The reason the first 20 to 25 requests appear correct is that early in the session, prior documents are a small fraction of the total context and the model’s attention favors the most recent (current) document. As the session grows, the signal-to-noise ratio degrades. By request 50, prior context occupies the majority of the context window, and the current document is a small fraction of what the model is conditioning on. At that point the contamination rate reaches a level the developer notices.

# Diagnosing the session state accumulation
# Confirm that MLCEngine holds history between calls

import mlc_llm

engine = mlc_llm.MLCEngine("Llama-3-8B-Instruct-q4f16_1-MLC")

# Step 1: Send a query containing a unique sentinel
response_1 = engine.chat.completions.create(
    messages=[{"role": "user", "content": "My document mentions XQRT-7749. What is it?"}],
    max_tokens=100,
    temperature=0.0,
)
print("Response 1:", response_1.choices[0].message.content)
# Output: "Based on your document, XQRT-7749 appears to be..."

# Step 2: Send a completely unrelated query — no mention of XQRT-7749
response_2 = engine.chat.completions.create(
    messages=[{"role": "user", "content": "What is the capital of France?"}],
    max_tokens=100,
    temperature=0.0,
)
print("Response 2:", response_2.choices[0].message.content)
# Output might include: "Paris. Also, regarding XQRT-7749 from your earlier document..."
# XQRT-7749 is in the response despite not being in query 2 — session contamination confirmed

# Step 3: Check if engine.reset() clears the state
engine.reset()

response_3 = engine.chat.completions.create(
    messages=[{"role": "user", "content": "What is the capital of France?"}],
    max_tokens=100,
    temperature=0.0,
)
print("Response 3 (after reset):", response_3.choices[0].message.content)
# Output: "The capital of France is Paris." — no reference to XQRT-7749
# engine.reset() cleared the session history — isolation confirmed

The fix: engine.reset() before each independent query

The fix is a single call to engine.reset() before each request. This clears the engine’s internal conversation history and KV cache state, ensuring that each query is processed with no context from prior queries. For multi-turn conversations within a single user session, the developer maintains explicit conversation history in application code and passes it in the messages array on each call, using a fresh engine state each time.

# qa_service.py — FIXED (Northfield MLC Inference Ltd.)
# mlc-llm 0.1.x, Python 3.11, MacBook Pro M2 Max 96GB

import mlc_llm
from fastapi import FastAPI

app = FastAPI()
engine = mlc_llm.MLCEngine("Llama-3-8B-Instruct-q4f16_1-MLC")


@app.post("/qa")
def answer_question(document: str, question: str) -> dict:
    prompt = f"""Answer the question using only information from the document below.
If the answer is not in the document, say "Not found in document."

Document:
{document}

Question: {question}"""

    # FIXED: reset engine state before each independent query
    # This clears the internal conversation history and KV cache
    # Each request is answered with no context from prior requests
    engine.reset()

    response = engine.chat.completions.create(
        messages=[{"role": "user", "content": prompt}],
        max_tokens=512,
        temperature=0.1,
    )

    return {"answer": response.choices[0].message.content}


# For multi-turn conversation use cases (where history SHOULD persist within a session):
# Option 1: Pass full history explicitly and reset between sessions
#
# def chat(session_history: list[dict], new_message: str) -> str:
#     engine.reset()
#     all_messages = session_history + [{"role": "user", "content": new_message}]
#     response = engine.chat.completions.create(
#         messages=all_messages,
#         max_tokens=512,
#     )
#     return response.choices[0].message.content
#
# Option 2: Use AsyncMLCEngine with one engine instance per active session
# (better for concurrent workloads — prevents shared state between sessions)

# After fix:
# Request 1–100+: each request answered with only current document as context ✓
# "Not found in document" correctly returned for out-of-scope questions ✓
# No cross-request contamination in 100-request isolation test ✓
# Response latency: 1.1s per request (engine.reset() adds ~8ms overhead) ✓

Validation

# Validation log — November 2026
# mlc-llm 0.1.x, Llama-3-8B INT4 W4A16, MacBook Pro M2 Max 96GB

# ---- Before fix (no engine.reset() — session accumulates across all requests) ----
# Requests 1–25:   contamination events: 0 (prior context < 10% of window)
# Requests 26–50:  contamination events: 3 (prior context ~40% of window)
# Requests 51–75:  contamination events: 11 (prior context ~65% of window)
# Requests 76–100: contamination events: 19 (prior context ~85% of window, then evicted)
# Total contamination rate: 33% across requests 50–100
# "Not found in document" false negative rate: 41%
#   (model fabricates answers from prior document context instead of returning not-found)

# ---- After fix (engine.reset() before each request) ----
# Sentinel contamination test: 0 contamination events across 100 requests ✓
#   (100 unique sentinel tokens; none appear in any subsequent response)
# "Not found in document" rate: 97% for out-of-scope questions ✓
#   (3% false negatives from model uncertainty on borderline cases — not contamination)
# engine.reset() latency overhead: 7.3ms per request (negligible)              ✓
# Total request latency (document 2,100 tokens, max_tokens=512): 1.14s          ✓

# ---- Hour breakdown ----
# Triage:     0.5h — contamination correlates with request count not content;
#             sentinel token test confirms MLCEngine holds prior session;
#             mlc-llm API docs confirm engine.reset() clears session history
# Root cause: 0.25h — MLCEngine designed for multi-turn chat; session accumulation
#             is intended behavior; missing reset() for stateless use case
# Fix:        0.5h — engine.reset() before each request handler; latency measured;
#             sentinel isolation test written and passing
# Validation: 0.25h — 100 requests; 0 contamination events; latency within SLA;
#             "Not found in document" false negative rate 3% (pre-existing, not regression)
# Total:      1.5h

# Retainer engagement cost: 1.5h (estimated $143–233 at mid-level rates)

Related retainer posts

Track retainer hours without a client portal

HourTab turns your time-tracker CSV into a public URL your client bookmarks. No login. No portal. CSV in, URL out.

See pricing →