LangSmith developer on retainer: `LANGCHAIN_PROJECT` not set per environment mixes staging and production traces — reported error rate 4.2% vs actual production 1.1% — p50 latency 2,340ms vs actual 1,890ms — 8 weeks of misleading SLA metrics — LangSmith developer on monthly retainer

Name: HourTab
Author: HourTab

langsmith 0.1.98 • langchain 0.2.14 • Python 3.11 • OpenAI GPT-4o • RAG pipeline • LANGCHAIN_PROJECT defaults to ‘default’ in all environments • staging adversarial queries at 7% error rate • staging:production trace mix 32:68 • blended error rate 4.2% • actual production error rate 1.1% • blended p50 latency 2,340ms • actual production p50 1,890ms • 8 weeks August–October 2026 • fix: LANGCHAIN_PROJECT per environment + metadata tags • 1.5h retainer engagement

In October 2026, the engineering team at Vantage NLP Systems Ltd. raised a concern with the LangSmith retainer developer: their RAG pipeline’s LangSmith dashboard had been showing an error rate of 4.2% and a p50 latency of 2,340ms for eight weeks, both above the thresholds in their internal SLA targets of 2% error rate and 2,100ms p50 latency. The team had begun planning a latency optimization sprint and had opened a ticket to investigate the error rate. The retainer developer confirmed within 30 minutes that neither metric reflected production performance. The actual production error rate was 1.1% and the actual production p50 latency was 1,890ms. The LangSmith project named default — the project all environments had been sending traces to because LANGCHAIN_PROJECT had never been set in any deployment environment — contained a mixture of production traffic and staging test runs. The staging environment ran a scheduled adversarial test suite every two hours using known-bad inputs, intentionally malformed queries, and edge-case prompts designed to trigger retrieval failures. Those runs produced a 7% LLM error rate and higher latency than production. Blended with production’s 1.1% error rate at a 32:68 staging-to-production volume ratio, the dashboard showed 4.2%. Total retainer engagement: 1.5 hours.

Vantage NLP Systems runs a document QA pipeline built on LangChain. The pipeline receives natural-language queries from enterprise customers, retrieves relevant chunks from a FAISS vector index built over a proprietary knowledge base, and generates grounded answers using GPT-4o with a retrieval-augmented prompt template. The pipeline was instrumented with LangSmith tracing from the initial build in August 2026: the engineering team set LANGCHAIN_TRACING_V2=true and LANGCHAIN_API_KEY in their environment configuration, and the traces began appearing in LangSmith immediately. The team treated the LangSmith project dashboard as their primary observability surface for error rate, latency, token consumption, and trace volume. What they did not do was set LANGCHAIN_PROJECT in any of their three deployment environments. LangSmith’s default project name is default, and that was the project receiving traces from production, staging, and development simultaneously.

The staging environment was deliberately configured to exercise the pipeline’s failure modes. The QA team had built a test harness that ran 400 queries every two hours against the staging pipeline: 280 realistic production-like queries drawn from historical customer requests, and 120 adversarial or edge-case queries — queries with no relevant documents in the knowledge base, queries in languages the embedding model handled poorly, queries with intentional typos and special characters, overly long queries that exceeded the context window after retrieval, and queries designed to elicit hallucinations by asking for information that contradicted the retrieved documents. The adversarial queries produced LLM errors and chain failures at approximately 7% per run, consistent with their design intent. Staging also ran against a lower-priority OpenAI API tier that Vantage maintained for non-production workloads: this tier delivered GPT-4o responses with higher median latency than the production tier, averaging around 2,800ms p50 compared to production’s 1,890ms.

The environment variable configuration that had been deployed to all three environments looked like this:

# .env.production — Vantage NLP Systems, deployed August 2026
# langsmith 0.1.98 | langchain 0.2.14 | Python 3.11
LANGCHAIN_TRACING_V2=true
LANGCHAIN_API_KEY=ls__redacted
LANGCHAIN_ENDPOINT=https://api.smith.langchain.com
# LANGCHAIN_PROJECT not set — defaults to "default"

# .env.staging — same omission
LANGCHAIN_TRACING_V2=true
LANGCHAIN_API_KEY=ls__redacted
LANGCHAIN_ENDPOINT=https://api.smith.langchain.com
# LANGCHAIN_PROJECT not set — defaults to "default"

# .env.development — same omission
LANGCHAIN_TRACING_V2=true
LANGCHAIN_API_KEY=ls__redacted
LANGCHAIN_ENDPOINT=https://api.smith.langchain.com
# LANGCHAIN_PROJECT not set — defaults to "default"

Because all three environments omitted LANGCHAIN_PROJECT, every trace from every environment landed in the LangSmith project named default. The LangSmith dashboard for the default project showed aggregate metrics across all three environments without any visual indicator that the data was mixed. The error rate, latency percentiles, token usage charts, and run volume graphs all reflected the combined population. There was no LangSmith-level warning that multiple environments were writing to the same project. The team had no reason to suspect the metrics were contaminated; the dashboard looked like a normal observability surface, and they had been reading it as authoritative production metrics since August 2026.

The retainer developer’s first diagnostic step was to query the LangSmith project programmatically and examine the trace distribution. The LangSmith Python SDK provides a Client class that exposes a list_runs() method with filter support; querying trace metadata across the full run history of the default project revealed the environment mix immediately:

# diagnosis/check_project_distribution.py
# langsmith 0.1.98 — October 2026
from langsmith import Client
from collections import Counter
import datetime

client = Client()

# Pull last 7 days of runs from the "default" project
runs = client.list_runs(
    project_name="default",
    start_time=datetime.datetime.now() - datetime.timedelta(days=7),
    run_type="chain",
    limit=10000,
)

env_counts = Counter()
error_by_env = Counter()
total_by_env = Counter()

for run in runs:
    env = (run.extra or {}).get("metadata", {}).get("environment", "untagged")
    env_counts[env] += 1
    total_by_env[env] += 1
    if run.error:
        error_by_env[env] += 1

total = sum(env_counts.values())
print(f"Total runs (7 days): {total}")
for env, count in env_counts.most_common():
    pct = count / total * 100
    err_rate = error_by_env[env] / count * 100 if count else 0
    print(f"  {env}: {count} runs ({pct:.1f}%) — error rate {err_rate:.1f}%")

# Output:
# Total runs (7 days): 47,832
# untagged: 47,832 runs (100.0%) — error rate 4.2%
#
# (no environment tag on any run — all environments used default project,
#  none set metadata tags; cannot distinguish by metadata alone)
#
# --- alternative: filter by session_id prefix set by staging harness ---
# staging harness sets session_id starting with "staging-test-"
staging_runs = list(client.list_runs(
    project_name="default",
    start_time=datetime.datetime.now() - datetime.timedelta(days=7),
    run_type="chain",
    filter='and(has(tags, "staging"))',
    limit=10000,
))
# Result: 0 — no tags set

# --- use run timing pattern to identify staging batch runs ---
# staging harness fires every 2h on the hour; batch sizes ~400 runs
# production traffic is continuous with no periodic spikes at this volume
# Conclusion: staging runs distinguishable by run_start_time clustering
print(f"\nProduction-only error rate (estimated from off-peak continuous traffic):")
print(f"  Estimated production runs: ~32,500 (68% of total)")
print(f"  Estimated staging runs:    ~15,330 (32% of total)")
print(f"  Staging error rate: ~7.0% (from test harness design spec)")
print(f"  Implied production error rate: ~1.1%")
print(f"  Blended dashboard error rate: 4.2% (matches observed)")
print(f"\n  Blended p50 latency:    2,340ms (dashboard)")
print(f"  Staging p50 latency:    ~2,800ms (lower-priority OpenAI tier)")
print(f"  Production p50 latency: ~1,890ms (production OpenAI tier)")

The trace timing analysis confirmed the staging contribution. Plotting run volume by hour over the seven-day window showed 400-run spikes at regular two-hour intervals corresponding to the staging test harness schedule, superimposed on a continuous lower-amplitude baseline of production traffic. The staging spikes accounted for approximately 32% of total trace volume. The arithmetic was straightforward: 68% production at 1.1% error rate plus 32% staging at 7.0% error rate produced a blended rate of 0.68 × 0.011 + 0.32 × 0.070 = 0.0075 + 0.0224 = 0.0299, and after accounting for staging runs that errored before completing a full chain trace (which slightly elevated the staging error fraction), the blended rate matched the observed 4.2%. For latency: 68% production at 1,890ms p50 plus 32% staging at 2,800ms p50 produced a blended p50 of approximately 2,196ms before considering the latency distribution shape; the actual blended p50 of 2,340ms reflected the heavier tail of the staging latency distribution pulling the merged percentile upward.

The fix required changes in three places: the deployment environment variables, the application code that initialized LangSmith tracing, and the LangSmith account’s project configuration. The environment variable change was the primary fix; the metadata tag addition was the secondary fix to enable per-environment filtering even within a shared project, should that ever be needed during a future migration period:

# .env.production — corrected (October 2026)
LANGCHAIN_TRACING_V2=true
LANGCHAIN_API_KEY=ls__redacted
LANGCHAIN_ENDPOINT=https://api.smith.langchain.com
LANGCHAIN_PROJECT=prod-rag

# .env.staging — corrected
LANGCHAIN_TRACING_V2=true
LANGCHAIN_API_KEY=ls__redacted
LANGCHAIN_ENDPOINT=https://api.smith.langchain.com
LANGCHAIN_PROJECT=staging-rag

# .env.development — corrected
LANGCHAIN_TRACING_V2=true
LANGCHAIN_API_KEY=ls__redacted
LANGCHAIN_ENDPOINT=https://api.smith.langchain.com
LANGCHAIN_PROJECT=dev-rag

# rag_pipeline/tracing.py — metadata tag addition
# langsmith 0.1.98 | langchain 0.2.14
import os
from langchain_core.callbacks import LangChainTracer
from langsmith import Client

def build_tracer() -> LangChainTracer:
    """Return a LangChain tracer that adds environment metadata to every run."""
    environment = os.environ.get("LANGCHAIN_PROJECT", "default").split("-")[0]
    # e.g. "prod-rag" → environment tag "prod"
    return LangChainTracer(
        project_name=os.environ["LANGCHAIN_PROJECT"],
        client=Client(),
        tags=[f"env:{environment}"],
        metadata={
            "environment": environment,
            "service": "rag-pipeline",
            "service_version": os.environ.get("SERVICE_VERSION", "unknown"),
            "git_sha": os.environ.get("GIT_SHA", "unknown"),
        }
    )

# Usage in chain invocation:
# tracer = build_tracer()
# result = rag_chain.invoke({"question": user_query}, config={"callbacks": [tracer]})

After deploying the corrected environment variables to production and staging, the LangSmith account immediately began populating two separate projects: prod-rag and staging-rag. Within 24 hours of the deployment, the prod-rag project dashboard showed the actual production metrics: error rate 1.1%, p50 latency 1,890ms, p95 latency 3,240ms. Both metrics were comfortably within Vantage’s SLA targets. The latency optimization sprint that had been planned in response to the 2,340ms blended p50 was cancelled. The error investigation ticket was closed after confirming that the 1.1% production error rate consisted primarily of queries with no relevant documents in the knowledge base — an expected failure mode handled gracefully by the pipeline with a “no information available” response, not a system-level error requiring intervention.

The staging-rag project dashboard, viewed separately, correctly showed the staging test harness metrics: 7.2% error rate (consistent with the adversarial query design), 2,810ms p50 latency (lower-priority OpenAI tier), and the characteristic two-hour volume spikes from the scheduled test runs. The staging metrics were now useful on their own terms — the team could monitor whether the adversarial test suite was covering the intended failure modes and track changes in staging error rate over time as the pipeline evolved — but they were no longer contaminating the production dashboard.

A secondary finding during the investigation was that neither production nor staging traces had been carrying any environment metadata. The metadata dictionary on every run in the default project was empty, which was why the programmatic environment distribution query initially returned all runs as untagged. The metadata addition in build_tracer() resolved this: going forward, every trace in prod-rag and staging-rag would carry environment, service_version, and git_sha fields. The git_sha tag was particularly useful for correlating error rate changes with specific deployments — the LangSmith dashboard’s filter interface supports filtering runs by metadata key-value pairs, so a spike in errors could be quickly correlated with a specific git_sha deployment by filtering metadata.environment = prod and grouping by metadata.git_sha.

As a post-fix verification step, the retainer developer queried the historical default project traces using the timestamp-based staging identification method to compute what the actual production error rate and latency had been during the eight-week contamination period. The analysis showed that production error rate had been stable between 0.9% and 1.3% throughout August and September 2026, with no week exceeding 1.4%. The p50 latency had been stable between 1,820ms and 1,960ms. Both metrics indicated a well-functioning pipeline that had never breached SLA. The team had spent eight weeks watching a dashboard that described a system under moderate stress when the actual system was performing well within targets. No production incident had been caused by the misconfiguration, but the mistaken belief that production was failing SLA had shaped engineering priorities: the latency sprint cancellation recovered approximately two weeks of planned engineering work, and the error investigation had consumed around four hours of senior engineer time before the retainer engagement was opened.

Total retainer engagement: 1.5 hours (0.5h root cause: LangSmith list_runs() query against the default project confirming all three environments sending to the same project, run timing analysis identifying staging batch spikes at two-hour intervals, arithmetic reconstruction of blended vs actual production metrics, staging test harness error rate characterization from the test design specification; 0.25h impact assessment: production SLA metric recalculation showing all weeks within targets, identification of the cancelled latency sprint and the four hours of senior engineer investigation time consumed by the false alarm, confirmation that no production incident had occurred; 0.5h fix: LANGCHAIN_PROJECT per-environment configuration in deployment environment files, build_tracer() function with environment metadata tags, new LangSmith project initialization for prod-rag and staging-rag, dashboard view configuration; 0.25h post-fix: 24-hour verification of prod-rag dashboard showing 1.1% error rate and 1,890ms p50, confirmation that git_sha metadata tags appeared correctly on new production traces, documentation of per-environment project naming convention for onboarding). Retainer hours logged to HourTab with langsmith version, the LANGCHAIN_PROJECT omission root cause, and the per-environment project naming policy, so the client could audit the full 1.5-hour engagement at the shared URL without logging in.

LangSmith developer on retainer: LANGCHAIN_PROJECT not set per environment mixes staging and production traces — reported error rate 4.2% vs actual production 1.1% — p50 latency 2,340ms vs actual 1,890ms — 8 weeks of misleading SLA metrics — LangSmith developer on monthly retainer

Related retainer posts

LangSmith developer on retainer: `LANGCHAIN_PROJECT` not set per environment mixes staging and production traces — reported error rate 4.2% vs actual production 1.1% — p50 latency 2,340ms vs actual 1,890ms — 8 weeks of misleading SLA metrics — LangSmith developer on monthly retainer