Opik developer on retainer: OPIK_PROJECT_NAME not set per environment — development and production traces share “Default Project” — LLM-as-judge scores dev traces with intentional errors — 68% reported correctness vs 88% production-only — 3 months of wasted RAG iteration — Opik developer on monthly retainer

opik 1.3.2 • Python 3.11 • LangChain 0.2.15 • OpenAI GPT-4o • FastAPI 0.111.0 • OPIK_PROJECT_NAME not set in either environment • development + production traces to “Default Project” • 200 dev traces/day at 10% correctness + 580 prod traces/day at 88% • mixed project shows 68% correctness • team iterates on RAG 3 months • fix: OPIK_PROJECT_NAME per environment • 88.3% production-only • 2.5h retainer engagement

Blueridge AI Consulting Ltd. builds a RAG-powered customer support assistant for mid-market SaaS companies. Their pipeline uses LangChain to retrieve relevant documentation chunks from a FAISS vector store and generates answers with GPT-4o. After three months of continuous iteration — adjusting system prompt phrasing, retrieval parameters, chunk size, and reranking thresholds — their Opik correctness evaluation dashboard remained stuck at 68%, well below the 75% threshold the product team had set as the quality gate for the next release. They had exhausted every explanation they could think of. The RAG pipeline itself was technically sound. They brought in an Opik retainer developer to diagnose why the metrics refused to improve regardless of the changes they made. The root cause was one missing environment variable, and it had been invisible for the entire three months.

Their Opik integration used the standard LangChain tracer pattern. The relevant code was identical across both their development and production deployments:

# app/tracer.py
# Blueridge AI Consulting Ltd.
# opik 1.3.2, Python 3.11, LangChain 0.2.15

from opik.integrations.langchain import OpikTracer

tracer = OpikTracer(
    tags=["rag-pipeline", "customer-support"],
    # No project_name argument here.
    # Reads from OPIK_PROJECT_NAME environment variable.
    # OPIK_PROJECT_NAME not set in either environment.
    # Both development and production trace to "Default Project".
)

# app/pipeline.py
from langchain.chains import RetrievalQA
from langchain_openai import ChatOpenAI
from langchain_community.vectorstores import FAISS

def build_pipeline(vectorstore: FAISS) -> RetrievalQA:
    llm = ChatOpenAI(model="gpt-4o", temperature=0)
    return RetrievalQA.from_chain_type(
        llm=llm,
        retriever=vectorstore.as_retriever(search_kwargs={"k": 4}),
        return_source_documents=True,
        callbacks=[tracer],  # Same tracer instance used in dev and production
    )

The environment files for the two deployments were:

# .env.development
OPIK_API_KEY=sk-opik-dev-xxxxxxxxxxxxxxxx
OPIK_WORKSPACE=blueridge-ai
# OPIK_PROJECT_NAME is not set — defaults to "Default Project"

# .env.production
OPIK_API_KEY=sk-opik-prod-xxxxxxxxxxxxxxxx
OPIK_WORKSPACE=blueridge-ai
# OPIK_PROJECT_NAME is not set — defaults to "Default Project"

# Both deployments use the same workspace, blueridge-ai.
# Both trace to "Default Project" within that workspace.

The development team had five engineers who ran the pipeline regularly throughout each workday. They used the development API endpoint for automated integration tests, regression tests after prompt changes, and manual exploratory testing. These test queries were not drawn from real user questions — they were purpose-built edge cases: inputs at the maximum token limit, completely off-topic questions like “RESET” and “!@#$%^&”, questions in languages not covered by the documentation corpus, intentionally ambiguous inputs designed to expose retrieval failures. The development pipeline’s answers to these inputs were wrong — by design. The point of the tests was to observe and handle failure modes correctly. The GPT-4o LLM-as-judge automated evaluator that the team had configured on “Default Project” in the Opik UI correctly identified these answers as wrong and scored them near zero. The evaluator was doing exactly what it was designed to do. The problem was the input it was receiving.

By the time the retainer developer was engaged, the Opik workspace had accumulated 47,230 traces in “Default Project” spanning July through October 2026. The retainer developer ran a diagnostic query against the Opik Python client to split those traces by origin:

# diagnosis/check_project_split.py
# Estimate dev vs production trace ratio in "Default Project".

import opik
import re

client = opik.Opik()

traces = client.get_traces_by_project(project_name="Default Project", limit=5000)

# Identify development traces by known test input patterns
DEV_PATTERNS = [
    r"^RESET",
    r"^\s*$",
    r"^!",
    r"^test_",
    r"^debug_",
    r"pytest",
    r"^[a-f0-9]{8}-",  # UUID test IDs passed as queries
]

dev_traces = []
prod_traces = []
for trace in traces:
    query_text = ""
    if trace.input and isinstance(trace.input, dict):
        query_text = str(trace.input.get("query", trace.input.get("question", "")))
    if any(re.search(p, query_text, re.IGNORECASE) for p in DEV_PATTERNS):
        dev_traces.append(trace)
    else:
        prod_traces.append(trace)

print(f"Sample size: {len(traces)} traces")
print(f"Dev traces (estimated): {len(dev_traces)} ({len(dev_traces)/len(traces)*100:.1f}%)")
print(f"Prod traces (estimated): {len(prod_traces)} ({len(prod_traces)/len(traces)*100:.1f}%)")

# Compute average evaluation score per segment
dev_scores = [t.feedback_scores[0].value for t in dev_traces if t.feedback_scores]
prod_scores = [t.feedback_scores[0].value for t in prod_traces if t.feedback_scores]

if dev_scores:
    print(f"Dev avg correctness: {sum(dev_scores)/len(dev_scores):.3f}")
if prod_scores:
    print(f"Prod avg correctness: {sum(prod_scores)/len(prod_scores):.3f}")

# Output:
# Sample size: 5000 traces
# Dev traces (estimated): 1285 (25.7%)
# Prod traces (estimated): 3715 (74.3%)
# Dev avg correctness: 0.097
# Prod avg correctness: 0.881

The result was unambiguous. Development traces made up 25.7% of the trace population with an average correctness of 0.097 — near zero, exactly as the development team intended. Production traces made up 74.3% with an average correctness of 0.881. The weighted average: 0.743 × 0.881 + 0.257 × 0.097 = 0.654 + 0.025 = 0.679 — matching almost exactly the 68% the dashboard had been reporting. The production correctness had been near 88% the entire time. Every prompt iteration and retrieval parameter adjustment the team had made over three months was reflected accurately in the production-only trace segment; those adjustments were just invisible in the polluted aggregate metric. The 75% threshold was never in question — production had already crossed it by month two.

The fix was a single line change to each environment file and the application’s startup configuration:

# .env.development — fixed
OPIK_API_KEY=sk-opik-dev-xxxxxxxxxxxxxxxx
OPIK_WORKSPACE=blueridge-ai
OPIK_PROJECT_NAME=customer-support-development  # ← Added

# .env.staging — fixed
OPIK_API_KEY=sk-opik-stg-xxxxxxxxxxxxxxxx
OPIK_WORKSPACE=blueridge-ai
OPIK_PROJECT_NAME=customer-support-staging  # ← Added

# .env.production — fixed
OPIK_API_KEY=sk-opik-prod-xxxxxxxxxxxxxxxx
OPIK_WORKSPACE=blueridge-ai
OPIK_PROJECT_NAME=customer-support-production  # ← Added

# Alternatively, set project_name directly in code using the ENVIRONMENT variable:
# tracer = OpikTracer(
#     project_name=f"customer-support-{os.getenv('ENVIRONMENT', 'development')}",
#     tags=["rag-pipeline", "customer-support"],
# )

After deploying the fix to all three environments and reconfiguring the automated evaluator to target customer-support-production exclusively, the Opik dashboard showed 88.3% correctness on the production project — above the 75% threshold the team had been trying to reach for three months. A new automated evaluator was also configured on customer-support-development with a separate rule designed for the development context: instead of penalizing intentional error-handling test outputs, the development evaluator checked only that error responses had the correct structured format. The two projects served two different evaluation purposes, and neither polluted the other.

The postmortem identified the mechanism by which this class of contamination goes undetected for so long. The Opik dashboard aggregate metric is correct — it accurately summarizes all traces in the project. There is no warning indicator in the Opik UI that a project contains traces from multiple environments or that evaluation scores may be biased by non-representative traces. The team had interpreted the 68% correctness as a true signal about their RAG pipeline quality and responded rationally by iterating on the pipeline. The rational response to a misleading signal is still the wrong response. The retainer developer added a startup-time assertion to the application code: if OPIK_PROJECT_NAME is unset or equals “Default Project”, the application raises a ConfigurationError and refuses to start. This ensured that no future deployment could accidentally pollute a shared project.

Total retainer engagement: 2.5 hours (0.5h initial triage: inspect Opik workspace, confirm “Default Project” is the only active project, ask team about development vs production traffic patterns, identify OPIK_PROJECT_NAME as the likely missing configuration; 1.0h diagnostic: write trace-splitting script, run against 5,000-trace sample, confirm 25.7% dev / 74.3% production split, compute per-segment correctness scores, confirm 88.1% production vs 9.7% development matches expected behavior from test input descriptions; 0.5h fix: update three environment files, redeploy, confirm “Default Project” trace count stops growing; 0.5h documentation: write postmortem, add OPIK_PROJECT_NAME assertion to startup, update engineering runbook).

Related retainer posts

Track retainer hours without a client portal

HourTab turns your time-tracker CSV into a public URL your client bookmarks. No login. No portal. CSV in, URL out.

See pricing →