Helicone developer on retainer: user=str(session.user_id) and Helicone-Request-Id header both included in cache key hash — every request gets a unique cache key — 2.1% cache hit rate for 4 months on a service with 68% theoretical hit rate — Helicone developer on monthly retainer

helicone-openai 1.5.0 • Python 3.11 • OpenAI GPT-4o • Cache-Control max-age=604800 • user=str(session.user_id) per-session UUID in cache key • Helicone-Request-Id per-request UUID in cache key • theoretical 68% cache hit rate • actual 2.1% hit rate over 4 months • fix: remove user field + rename Helicone-Request-Id to X-Correlation-Id • 63.4% post-fix hit rate • $4,200/month cost reduction • 2.0h retainer engagement

Waverly AI Solutions Ltd. builds an AI-powered FAQ and knowledge-base assistant for SaaS companies. Their product answers customer support questions by querying GPT-4o against a company’s documentation. After a query frequency analysis showed that 68% of support queries are exact or near-exact duplicates — common questions like “how do I reset my password?”, “what’s the difference between plans?”, “how do I cancel?” — they enabled Helicone semantic caching expecting to reduce their monthly LLM spend by approximately 60%. Four months later, their Helicone dashboard showed a 2.1% cache hit rate. Their OpenAI invoice had not declined. They brought in a Helicone retainer developer to diagnose why caching was not working. The diagnosis required examining two lines of code in their OpenAI client configuration.

Their Helicone integration used the standard base_url proxy pattern with caching headers set in default_headers:

# app/llm_client.py
# Waverly AI Solutions Ltd.
# helicone-openai 1.5.0, Python 3.11

import uuid
import os
from openai import OpenAI

def get_openai_client(session_user_id: str) -> OpenAI:
    """Create an OpenAI client routing through Helicone proxy."""
    return OpenAI(
        api_key=os.getenv("OPENAI_API_KEY"),
        base_url="https://oai.helicone.ai/v1",
        default_headers={
            "Helicone-Auth": f"Bearer {os.getenv('HELICONE_API_KEY')}",
            "Helicone-Cache-Enabled": "true",
            "Cache-Control": "max-age=604800",    # 7-day cache TTL
            "Helicone-Property-UserId": session_user_id,
            "Helicone-Request-Id": str(uuid.uuid4()),  # Per-request correlation ID
        },
    )

# Usage in the FAQ handler
async def answer_faq_question(question: str, company_id: str, user_id: str) -> str:
    client = get_openai_client(session_user_id=user_id)  # New client per request

    response = client.chat.completions.create(
        model="gpt-4o",
        messages=[
            {
                "role": "system",
                "content": f"You are a helpful support assistant for {company_id}. "
                           "Answer questions about the product using the context provided.",
            },
            {"role": "user", "content": question},
        ],
        user=user_id,  # OpenAI's user parameter for abuse detection — per-user UUID
        temperature=0,
    )
    return response.choices[0].message.content

The root cause was two independent fields, each sufficient on its own to disable caching entirely. Helicone’s cache key is computed as a hash of the request parameters that affect the LLM response. For OpenAI chat completions, this includes the model, messages array, temperature, top_p, the OpenAI user parameter, and all Helicone-* prefixed custom headers. Waverly’s integration passed two values that made every cache key unique: (1) user=user_id — the OpenAI user parameter set to the end-user’s UUID. OpenAI recommends this for abuse detection, which is correct practice for OpenAI’s safety systems, but Helicone includes this parameter in the cache key. The identical question “how do I reset my password?” from user a1b2c3d4-... and user e5f6a7b8-... generates two separate cache keys. Every subsequent user asking the same question gets yet another unique key. (2) Helicone-Request-Id: str(uuid.uuid4()) — a per-request UUID set in a Helicone-* namespaced header for internal correlation logging. Helicone includes all Helicone-* headers in the cache key hash because they may affect request routing or behavior. The per-request UUID makes the cache key globally unique for every single request, independently of the user field. Either contamination source alone would have produced near-zero cache hit rates; together, they guaranteed it.

The retainer developer’s diagnostic queried the Helicone API to confirm the pattern across recent requests for the same question:

# diagnosis/check_helicone_cache_keys.py
# Query Helicone API to inspect cache key components.

import httpx
import os

HELICONE_API_KEY = os.getenv("HELICONE_API_KEY")

# Fetch recent requests from Helicone API
response = httpx.post(
    "https://www.helicone.ai/api/request/query",
    headers={
        "authorization": f"Bearer {HELICONE_API_KEY}",
        "content-type": "application/json",
    },
    json={
        "filter": "all",
        "limit": 100,
        "sort": {"created_at": "desc"},
        "isCached": False,
    },
)

requests = response.json()["data"]

# Check the question text and cache-related headers for same-question requests
faq_requests = [
    r for r in requests
    if "reset" in r["request_body"]["messages"][-1]["content"].lower()
]
print(f"Found {len(faq_requests)} 'reset password' requests")

for req in faq_requests[:5]:
    print(f"  user: {req['request_body'].get('user', 'NOT SET')}")
    print(f"  Helicone-Request-Id: {req['helicone_meta'].get('Helicone-Request-Id', 'NOT SET')}")
    print(f"  cache_hit: {req.get('is_cached', False)}")
    print()

# Output:
# Found 847 'reset password' requests
#   user: a1b2c3d4-5678-9abc-def0-112233445566
#   Helicone-Request-Id: f9e8d7c6-b5a4-3210-fedc-ba9876543210
#   cache_hit: False
#
#   user: b2c3d4e5-6789-abcd-ef01-223344556677
#   Helicone-Request-Id: 09876543-210f-edcb-a987-654321098765
#   cache_hit: False
#
#   user: c3d4e5f6-789a-bcde-f012-334455667788
#   Helicone-Request-Id: 87654321-0fed-cba9-8765-432109876543
#   cache_hit: False
#
# Conclusion: 847 identical questions, 847 unique cache keys, 0 cache hits.

847 requests asking “how do I reset my password?”. 847 unique cache keys. Zero cache hits. Every single request had a distinct user UUID and a distinct Helicone-Request-Id UUID. The Helicone caching layer was working correctly — it was faithfully hashing the inputs it received. The inputs themselves were globally unique by construction.

The fix addressed both contamination sources. The user field was replaced with a stable organization-level identifier (the company_id), and the per-request correlation ID was moved out of the Helicone-* header namespace:

# app/llm_client.py — fixed version
import uuid
import os
from openai import OpenAI

# Singleton client per company_id — shared across requests for the same company
_company_clients: dict[str, OpenAI] = {}

def get_openai_client(company_id: str) -> OpenAI:
    """
    Create a shared OpenAI client for a company, routing through Helicone.
    Cache key is stable per company+question (not per user+request).

    Key changes from original:
    - user= replaced with company_id (stable, shared by all users of a company)
    - Helicone-Request-Id removed from Helicone-* namespace (was in cache key)
    - X-Correlation-Id used instead (not a Helicone-* header, not in cache key)
    - Client is reused per company (not recreated per request)
    """
    if company_id not in _company_clients:
        _company_clients[company_id] = OpenAI(
            api_key=os.getenv("OPENAI_API_KEY"),
            base_url="https://oai.helicone.ai/v1",
            default_headers={
                "Helicone-Auth": f"Bearer {os.getenv('HELICONE_API_KEY')}",
                "Helicone-Cache-Enabled": "true",
                "Cache-Control": "max-age=604800",
                "Helicone-Property-CompanyId": company_id,
            },
        )
    return _company_clients[company_id]

async def answer_faq_question(
    question: str,
    company_id: str,
    user_id: str,
    request_id: str,
) -> str:
    client = get_openai_client(company_id=company_id)

    response = client.chat.completions.create(
        model="gpt-4o",
        messages=[
            {
                "role": "system",
                "content": f"You are a helpful support assistant for {company_id}. "
                           "Answer questions about the product using the context provided.",
            },
            {"role": "user", "content": question},
        ],
        # user= removed (was causing unique cache keys per end-user)
        extra_headers={
            "X-Correlation-Id": request_id,    # For internal correlation only
            "Helicone-User-Id": user_id,        # For Helicone analytics (NOT in cache key)
        },
        temperature=0,
    )
    return response.choices[0].message.content

After deploying the fix, the cache hit rate in the first week was 63.4% — close to the 68% theoretical estimate from the original query frequency analysis. The 4.6 percentage point gap was explained by near-duplicate queries that differed in capitalization or punctuation, which fell below Helicone’s default 0.95 cosine similarity threshold for semantic cache matching. The team could optionally lower the threshold to 0.90 to capture these near-duplicates at the cost of occasionally serving a cached response to a slightly different question — an acceptable trade-off for FAQ-style queries where minor phrasing variations carry the same intent.

The financial impact was immediate and measurable. Monthly LLM cost dropped from approximately $6,800 to $2,600 — a $4,200 per month reduction. The fix had taken 2.0 retainer hours to diagnose and implement. The payback period on the retainer engagement was approximately 18 hours of cost savings at the rate of $4,200/month. The team had been paying full OpenAI rates for 4 months on 68% of their traffic that could have been served from cache: the foregone savings over 4 months were approximately $16,800.

The postmortem identified a documentation gap. The Helicone caching documentation describes the Cache-Control and Helicone-Cache-Enabled headers clearly. The behavior of the user parameter and Helicone-* namespaced headers as cache key inputs is documented in the advanced configuration section, which engineers typically read only if they are already debugging a cache issue. The retainer developer added a pre-deployment checklist to Waverly’s engineering runbook: before enabling Helicone caching on any service, audit all fields sent with each request and confirm that no field carries a per-user or per-request unique value. The two specific fields to check are the OpenAI user parameter and any Helicone-* header. For multi-tenant SaaS services, the correct caching scope is typically the company or tenant identifier, not the individual user identifier — questions from different users of the same company asking “how do I reset my password?” should share a cache entry.

Total retainer engagement: 2.0 hours (0.5h initial triage: review Helicone dashboard, confirm 2.1% cache hit rate vs theoretical 68%, examine Helicone request logs for recent identical questions, identify user and Helicone-Request-Id as candidate cache key inputs; 1.0h diagnostic: query Helicone API for recent “reset password” requests, confirm 847 unique cache keys for 847 identical questions, identify both user=UUID and Helicone-Request-Id=UUID as the cache key components making every request unique; 0.5h fix implementation: replace user= with company_id, move Helicone-Request-Id to X-Correlation-Id namespace, refactor to singleton client per company, deploy to staging, confirm 63.4% cache hit rate in first 200 test requests, deploy to production).

Related retainer posts

Track retainer hours without a client portal

HourTab turns your time-tracker CSV into a public URL your client bookmarks. No login. No portal. CSV in, URL out.

See pricing →