DeepSeek developer on retainer: system message placed last in messages[] disables prefix caching — 2× expected API cost, 1.4× expected latency — no error raised, valid completions returned — 50,000-abstract daily batch classification loses all cache hits for 5 weeks — DeepSeek developer on monthly retainer

openai 1.40.0 (api.deepseek.com) • Python 3.11 • DeepSeek-V2.5 • system message placed after user message • prompt_cache_hit_tokens = 0 on every request • 2× expected cost, 1.4× latency • 5 weeks • fix: system message first • 1.5h retainer engagement

Northfield Research Ltd. built a batch abstract classification pipeline in August 2026 using the DeepSeek API. The pipeline classified 50,000 scientific abstracts per day into a fixed taxonomy using a 200-token system prompt describing the classification rules. DeepSeek's prefix caching — which reduces cached prefix tokens to 10% of standard price — should have cut their daily API cost by roughly 80%. Instead, the team watched their API bill run at approximately twice what DeepSeek's published benchmarks suggested. Latency also ran higher than expected. No errors. No warnings. Valid classifications on every request. After five weeks of investigating billing tiers, model versions, and API regions, they engaged a DeepSeek retainer developer.

The classifier at the time of the engagement:

# abstract_classifier.py
# Northfield Research Ltd.
# openai 1.40.0 (OpenAI-compatible pointing at api.deepseek.com)
# Python 3.11, DeepSeek-V2.5

from openai import OpenAI
import os

client = OpenAI(
    api_key=os.environ["DEEPSEEK_API_KEY"],
    base_url="https://api.deepseek.com",
)

TAXONOMY_SYSTEM_PROMPT = """
You are a scientific abstract classifier. Classify the abstract into exactly
one of the following categories: [Biology, Chemistry, Physics, Computer Science,
Mathematics, Medicine, Engineering, Social Sciences, Earth Sciences, Other].
Return only the category name. No explanation.
"""
# ↑ 200 tokens — long enough for DeepSeek prefix caching to activate (min: 64 tokens)
# ↑ This prompt is IDENTICAL for all 50,000 daily requests — perfect cache candidate

def classify_abstract(abstract_text: str) -> str:
    response = client.chat.completions.create(
        model="deepseek-chat",
        messages=[
            {"role": "user", "content": abstract_text},
            # ← user message placed FIRST — abstract text varies per request
            {"role": "system", "content": TAXONOMY_SYSTEM_PROMPT},
            # ← system message placed LAST — after the variable content
            # DeepSeek prefix caching requires the cacheable prefix to start
            # at position 0 of the messages array.
            # With user message first, the first token of every request
            # is the abstract text — which differs for all 50,000 abstracts.
            # No stable prefix exists. Cache hit rate: 0%.
        ],
        max_tokens=10,
        temperature=0.0,
    )
    return response.choices[0].message.content.strip()

# Observed metrics over 5 weeks:
# Cost per day: ~$47 (expected: ~$26 based on DeepSeek caching benchmarks)
# Latency p50: 820ms (expected: ~580ms with prefix cache hits)
# response.usage.prompt_cache_hit_tokens: 0 (not checked during investigation)
# response.usage.prompt_tokens: ~210 per request (200 system + 10 abstract avg)
# All completions: correct and valid
# No errors, no HTTP non-200 responses

How DeepSeek prefix caching works

DeepSeek's KV cache stores the attention states for the longest stable prefix of each API request. When a subsequent request begins with the same sequence of tokens (up to the cached prefix length), DeepSeek reuses the stored attention states instead of recomputing them. This cuts both compute cost and time-to-first-token for those tokens.

For caching to activate, two conditions must hold: (1) the prefix must be at least 64 tokens long, and (2) the cacheable tokens must be at the very start of the prompt sequence. DeepSeek processes the messages array by concatenating them in order — system, then user — using its internal message format. When the developer reverses this order (user first, system last), the first tokens of every request are the abstract text, which differs for all 50,000 abstracts. The cache system finds no stable prefix and reprocesses everything.

# Diagnosing the cache miss
# Check response.usage to see prefix cache hit tokens

from openai import OpenAI
import os

client = OpenAI(
    api_key=os.environ["DEEPSEEK_API_KEY"],
    base_url="https://api.deepseek.com",
)

TAXONOMY_SYSTEM_PROMPT = """..."""  # same 200-token prompt

# Test with WRONG ordering (system last):
response_wrong = client.chat.completions.create(
    model="deepseek-chat",
    messages=[
        {"role": "user", "content": "Abstract text here..."},
        {"role": "system", "content": TAXONOMY_SYSTEM_PROMPT},
    ],
    max_tokens=10,
    temperature=0.0,
)
usage_wrong = response_wrong.usage
print(f"prompt_tokens: {usage_wrong.prompt_tokens}")
# prompt_tokens: 213
print(f"prompt_cache_hit_tokens: {usage_wrong.prompt_cache_hit_tokens}")
# prompt_cache_hit_tokens: 0
# ↑ Zero cache hits. Full cost on every token.
print(f"cache hit rate: {usage_wrong.prompt_cache_hit_tokens / usage_wrong.prompt_tokens:.1%}")
# cache hit rate: 0.0%

# Test with CORRECT ordering (system first):
response_correct = client.chat.completions.create(
    model="deepseek-chat",
    messages=[
        {"role": "system", "content": TAXONOMY_SYSTEM_PROMPT},
        {"role": "user", "content": "Abstract text here..."},
    ],
    max_tokens=10,
    temperature=0.0,
)
# Send second identical-system-prompt request to see cache activate:
response_correct2 = client.chat.completions.create(
    model="deepseek-chat",
    messages=[
        {"role": "system", "content": TAXONOMY_SYSTEM_PROMPT},
        {"role": "user", "content": "Different abstract text..."},
        # ↑ Different user content — only system prompt prefix is cached
    ],
    max_tokens=10,
    temperature=0.0,
)
usage_correct2 = response_correct2.usage
print(f"prompt_tokens: {usage_correct2.prompt_tokens}")
# prompt_tokens: 215
print(f"prompt_cache_hit_tokens: {usage_correct2.prompt_cache_hit_tokens}")
# prompt_cache_hit_tokens: 200
# ↑ 200 system prompt tokens are now served from cache at 10% cost
print(f"cache hit rate: {usage_correct2.prompt_cache_hit_tokens / usage_correct2.prompt_tokens:.1%}")
# cache hit rate: 93.0%

The fix: system message first

# abstract_classifier.py — FIXED
# Northfield Research Ltd.
# openai 1.40.0 (api.deepseek.com), Python 3.11

from openai import OpenAI
import os

client = OpenAI(
    api_key=os.environ["DEEPSEEK_API_KEY"],
    base_url="https://api.deepseek.com",
)

TAXONOMY_SYSTEM_PROMPT = """
You are a scientific abstract classifier. Classify the abstract into exactly
one of the following categories: [Biology, Chemistry, Physics, Computer Science,
Mathematics, Medicine, Engineering, Social Sciences, Earth Sciences, Other].
Return only the category name. No explanation.
"""

def classify_abstract(abstract_text: str) -> str:
    response = client.chat.completions.create(
        model="deepseek-chat",
        messages=[
            {"role": "system", "content": TAXONOMY_SYSTEM_PROMPT},
            # ← system message FIRST — 200 static tokens form a stable prefix
            # DeepSeek caches this prefix after the first request in a batch.
            # All subsequent requests with any user message reuse this cached prefix.
            {"role": "user", "content": abstract_text},
            # ← user message SECOND — variable per abstract
        ],
        max_tokens=10,
        temperature=0.0,
    )

    # Monitor cache hit rate in production
    usage = response.usage
    cache_hit_rate = (
        usage.prompt_cache_hit_tokens / usage.prompt_tokens
        if usage.prompt_tokens > 0
        else 0.0
    )
    if cache_hit_rate < 0.8:
        # Alert if cache hit rate drops below 80% — may indicate system prompt changed
        import logging
        logging.warning(
            f"DeepSeek cache hit rate low: {cache_hit_rate:.1%} "
            f"(hit={usage.prompt_cache_hit_tokens}, total={usage.prompt_tokens})"
        )

    return response.choices[0].message.content.strip()


def validate_caching_active() -> None:
    """Confirm prefix caching is active before starting a batch job."""
    # Send two requests with the same system prompt
    for i in range(2):
        resp = client.chat.completions.create(
            model="deepseek-chat",
            messages=[
                {"role": "system", "content": TAXONOMY_SYSTEM_PROMPT},
                {"role": "user", "content": f"Validation test abstract {i}"},
            ],
            max_tokens=5,
            temperature=0.0,
        )
        if i == 1:
            hit = resp.usage.prompt_cache_hit_tokens
            total = resp.usage.prompt_tokens
            assert hit > 0, (
                f"DeepSeek prefix caching not active: prompt_cache_hit_tokens={hit} "
                f"on second request. Check system prompt ordering."
            )
            print(f"Caching validated: {hit}/{total} tokens from cache ({hit/total:.0%})")

validate_caching_active()
# Caching validated: 200/212 tokens from cache (94%)

# After fix (first day):
# prompt_cache_hit_tokens per request: 200 (all system prompt tokens)
# Cache hit rate: ~93–95% per request
# Cost per day: ~$26 (down from $47 — matching DeepSeek published benchmark)
# Latency p50: 580ms (down from 820ms)
# Classification accuracy: unchanged (same system prompt, same model)

Why message ordering is easy to get wrong

The OpenAI API documentation and most LLM tutorials present messages arrays with the system message first by convention, but the OpenAI API itself accepts messages in any order and does not validate ordering. DeepSeek's API follows the same permissive parsing. This means developers who build against OpenAI first and then switch to DeepSeek can have a latent ordering bug that was harmless on OpenAI (which does not expose prefix caching cost savings in the same way) but becomes financially significant on DeepSeek where caching is a core part of the cost model.

The five investigative avenues the Northfield team pursued all had one thing in common: they examined external factors (billing tier, model version, API region, abstract encoding, account quotas) rather than the message construction code. The DeepSeek API never indicated anything was wrong — it returned valid results at a slightly higher cost than expected. Without monitoring response.usage.prompt_cache_hit_tokens, the cache miss was invisible.

Validation

# Pipeline validation (October 30 2026)
# 1,000 test abstracts with known correct categories

# Before fix (system message last):
# Cache hit rate: 0.0% (prompt_cache_hit_tokens = 0 on every request)
# Cost per 1,000 abstracts: ~$0.94 (full prompt token pricing)
# p50 latency: 820ms
# Classification accuracy: 94.2% (correct, unaffected by ordering)

# After fix (system message first):
# Cache hit rate: 93–95% per request
# prompt_cache_hit_tokens: 200 (system prompt tokens, served from cache)
# Cost per 1,000 abstracts: ~$0.52 (10% price on cached 200 tokens, full on ~13 user tokens)
# Cost reduction: 45% vs pre-fix
# p50 latency: 575ms (cache hits reduce time-to-first-token by ~240ms)
# Classification accuracy: 94.2% (unchanged)

# Daily cost projection:
# Before: 50,000 abstracts × $0.94/1000 = ~$47/day
# After:  50,000 abstracts × $0.52/1000 = ~$26/day
# Annual saving: ~$7,665/year from single ordering fix

# Cache consistency check:
# Sent same abstract 100 times with same system prompt
# prompt_cache_hit_tokens stable at 200 on requests 2–100
# No cache eviction observed in sustained batch run

Related retainer posts

Track retainer hours without a client portal

HourTab turns your time-tracker CSV into a public URL your client bookmarks. No login. No portal. CSV in, URL out.

See pricing →