OpenLIT developer on retainer: BatchSpanProcessor default 5,000ms flush interval exceeds AWS Lambda execution window — openlit 1.22.0 — 91% of LLM traces dropped for 5 months — $10,900/month in OpenAI costs invisible in dashboard — OpenLIT developer on monthly retainer

openlit 1.22.0 • Python 3.11 • AWS Lambda • LangChain 0.2.15 • OpenAI Python SDK 1.35.0 • OTLP/HTTP to Grafana Tempo • BatchSpanProcessor schedule_delay_millis=5000 • Lambda container freezes before flush • 91% of LLM spans queued but never exported • 5 months May–October 2026 • $10,900/month invisible OpenAI costs • fix: provider.force_flush() + disable_batch=True • 2.5h retainer engagement

Meridian Analytics Corp. runs a contract document analysis pipeline as a series of AWS Lambda functions. Each invocation receives a document from an S3 event, processes it through a LangChain + GPT-4o pipeline that extracts entities, classifies risk categories, and generates a structured summary, then writes the output to DynamoDB. The Lambda functions run at a rate of roughly 60 invocations per hour during business hours. Meridian uses OpenLIT for LLM observability — monitoring token consumption, model costs, and latency distributions across their extraction pipeline. In May 2026 they integrated openlit 1.22.0 by calling openlit.init() at module level in their Lambda handler. By October 2026, their finance team noticed a persistent $10,900/month discrepancy between OpenAI billing and the costs attributed in the OpenLIT dashboard. OpenAI billing consistently showed $12,000/month. OpenLIT showed $1,080/month. Meridian’s engineering team assumed the other $10,900 was coming from a different application or a service account they hadn’t tracked down yet. A DevOps engineer spent two weeks investigating OTLP network connectivity between Lambda and Grafana Tempo before a retainer developer identified the actual root cause in a two-minute code review.

The Lambda handler was structured like this:

# lambda_function.py
# Meridian Analytics Corp.
# openlit 1.22.0, Python 3.11, AWS Lambda

import openlit
from langchain_openai import ChatOpenAI
from langchain.chains import LLMChain
from langchain.prompts import ChatPromptTemplate
import boto3
import json

# Module-level init — runs once on Lambda cold start
openlit.init(
    otlp_endpoint="http://tempo.internal.meridian.io:4318/v1/traces",
    environment="production",
    application_name="doc-analysis",
)

dynamodb = boto3.resource("dynamodb")
table = dynamodb.Table("DocumentAnalysis")


def lambda_handler(event, context):
    """Process a single contract document from S3."""

    # Extract document content from S3 event
    s3_key = event["Records"][0]["s3"]["object"]["key"]
    content = download_from_s3(s3_key)

    # Run LangChain pipeline
    llm = ChatOpenAI(model="gpt-4o", temperature=0)
    prompt = ChatPromptTemplate.from_messages([
        ("system", "Extract entities and risk categories from this contract."),
        ("human", "{content}")
    ])
    chain = LLMChain(llm=llm, prompt=prompt)
    result = chain.invoke({"content": content})

    # Store output
    table.put_item(Item={
        "document_id": s3_key,
        "analysis": result["text"],
    })

    return {"statusCode": 200, "body": json.dumps({"document_id": s3_key})}

The problem is in how AWS Lambda manages process execution after the handler returns. When lambda_handler returns the {"statusCode": 200, ...} response, Lambda immediately freezes the execution environment — it suspends the entire process and does not give it CPU time until the next invocation arrives. OpenLIT’s default span processor is the OpenTelemetry BatchSpanProcessor. The BatchSpanProcessor works by accumulating completed spans in an in-memory queue and flushing that queue to the configured OTLP exporter on a background thread that fires every schedule_delay_millis milliseconds — 5,000ms by default. The background flush thread is a standard Python daemon thread: it sits in a loop, calls time.sleep(5.0), wakes up, exports the accumulated spans, then sleeps again. This works correctly in long-running servers where the process stays alive continuously. In Lambda, the timeline is:

  1. Lambda cold start: openlit.init() runs, BatchSpanProcessor background thread starts, sleeps for 5 seconds.
  2. Lambda invokes handler: LangChain + GPT-4o call executes in ~2.2 seconds, spans are added to the BatchSpanProcessor queue.
  3. Handler returns response at ~2.2 seconds into the invocation.
  4. Lambda freezes the process immediately. The background thread is mid-sleep at its 5-second timer.
  5. Lambda thaws the process on the next invocation. The background thread resumes from its frozen position.
  6. If the next invocation arrives within the remaining ~2.8 seconds of the timer, the spans from the previous invocation are still in the queue when the new LangChain call starts adding its own spans.
  7. Eventually the 5-second timer fires and the batch is flushed — but only if the execution environment stays alive long enough for the timer to complete.

The critical case is the final invocation before the execution environment is terminated. Lambda destroys idle execution environments after approximately 15 minutes of inactivity (the exact duration varies and is not guaranteed). For Meridian’s pipeline, invocations arrived in bursts during business hours and then went idle at night. Every night, Lambda destroyed the execution environments after the last daytime invocation. The spans from those final invocations — and from any invocations where the environment was recycled mid-day due to Lambda’s internal load balancing — were queued in the BatchSpanProcessor but never exported. Over five months, roughly 91% of all invocations ended with span loss.

The retainer developer confirmed the diagnosis by adding a single diagnostic log line to the handler, using the OpenTelemetry SDK’s internal access to the BatchSpanProcessor queue:

# Diagnostic: check BatchSpanProcessor queue depth after handler completes
from opentelemetry import trace
from opentelemetry.sdk.trace.export import BatchSpanProcessor

provider = trace.get_tracer_provider()

# Access the BatchSpanProcessor to inspect queue depth
for span_processor in provider._active_span_processor._span_processors:
    if isinstance(span_processor, BatchSpanProcessor):
        queue_size = span_processor._queue.qsize()
        print(f"BatchSpanProcessor queue depth after handler: {queue_size}")
        # Output: BatchSpanProcessor queue depth after handler: 4
        # (4 spans from the LangChain + GPT-4o call, not yet exported)

The queue contained four spans after the handler completed: one for the LangChain LLMChain invocation, one for the ChatOpenAI call, one for the chat completion request, and one for the token counting operation. All four were waiting for the 5-second flush timer. None of them would be exported before Lambda froze the process.

The fix was a two-step change. First, add an explicit force_flush() call at the end of the handler to synchronously export all queued spans before returning:

# Step 1: force_flush() before handler return
from opentelemetry import trace

def lambda_handler(event, context):
    s3_key = event["Records"][0]["s3"]["object"]["key"]
    content = download_from_s3(s3_key)

    llm = ChatOpenAI(model="gpt-4o", temperature=0)
    prompt = ChatPromptTemplate.from_messages([...])
    chain = LLMChain(llm=llm, prompt=prompt)
    result = chain.invoke({"content": content})

    table.put_item(Item={"document_id": s3_key, "analysis": result["text"]})

    # Flush all queued spans before Lambda freezes the process
    provider = trace.get_tracer_provider()
    provider.force_flush(timeout_millis=10_000)

    return {"statusCode": 200, "body": json.dumps({"document_id": s3_key})}

force_flush(timeout_millis=10_000) blocks the current thread until all spans in the queue have been exported to the OTLP endpoint, or until the 10-second timeout expires. This adds the OTLP network round-trip (typically 15–40ms to Grafana Tempo on the same VPC) to each Lambda invocation’s execution time. For a pipeline where each invocation already takes 2–3 seconds waiting for GPT-4o, an additional 30ms is negligible.

The second fix was to reconfigure openlit.init() with disable_batch=True to make the behavior permanent and not dependent on remembering to call force_flush() in every handler function:

# Step 2: disable batching permanently for Lambda
openlit.init(
    otlp_endpoint="http://tempo.internal.meridian.io:4318/v1/traces",
    environment="production",
    application_name="doc-analysis",
    disable_batch=True,   # Use SimpleSpanProcessor: exports each span synchronously
)

With disable_batch=True, OpenLIT uses a SimpleSpanProcessor internally. The SimpleSpanProcessor calls exporter.export([span]) synchronously every time a span ends — no background thread, no queue, no buffering. Each span is exported immediately to Grafana Tempo within the span’s own execution context. There is no race condition with the Lambda freeze. The force_flush() call in the handler was also kept as a defensive measure.

After deploying the fix, the first hour of invocations produced 59 spans in Grafana Tempo — one complete LangChain trace per invocation, for 59 of the 60 invocations in that hour (the one missed was a Lambda cold start during the deployment rollout). The OpenLIT cost dashboard immediately showed $195 attributed in the first hour, consistent with the OpenAI billing API’s record for the same window. Extrapolated to a full month, the $1,080/month figure would become $12,000/month — exactly matching the OpenAI billing. The $10,900 “unaccounted” spend had been attributed to OpenAI all along; it was just invisible in OpenLIT due to span loss.

The retainer developer ran a retroactive reconciliation by querying five months of OpenAI usage logs from the OpenAI API’s usage endpoint and comparing them against the OpenLIT span records for the same period. The reconciliation confirmed that 91.2% of API calls — 5 months × ~43,200 invocations per month × 91.2% = ~197,000 uninstrumented invocations — had produced no spans in OpenLIT. The model cost breakdown by invocation that OpenLIT was supposed to provide — cost per document, cost per risk category classification, latency distribution by document length — had been based on the 8.8% of invocations that happened to export spans before the Lambda environment was recycled. Those happened to be the invocations that ran in quick succession during the highest-traffic windows, producing execution environments that stayed warm long enough for the 5-second flush timer to fire. The 8.8% sample was not representative. Cost-per-document estimates derived from OpenLIT data had been off by 10.4x.

The postmortem resulted in one new architectural standard for Meridian’s Lambda functions: any Lambda function that uses an OpenTelemetry-based SDK — not just OpenLIT, but also any other framework that instruments via the OTel SDK — must be tested by running 20 invocations in isolation, with 60-second gaps between each invocation to simulate environment recycling, and confirming that all 20 invocations produce spans in the observability backend. The test must run as a post-deploy smoke test in their CI/CD pipeline before any Lambda function is promoted to the production environment. The 60-second gap between invocations is designed to exceed Lambda’s typical warm environment timeout for low-traffic functions, forcing each test invocation to simulate a final-invocation-before-termination scenario.

Total retainer engagement: 2.5 hours (0.5h initial triage: query Grafana Tempo for span counts by Lambda function name for the past 30 days; compare against CloudWatch invocation metrics for the same period; confirm 91% gap; rule out OTLP network connectivity by checking Tempo ingest metrics for any rejected spans; 0.5h root cause: review lambda_function.py module-level openlit.init() call; identify BatchSpanProcessor as default span processor; confirm schedule_delay_millis=5000 default; read AWS Lambda execution model documentation confirming process freeze on handler return; add queue depth diagnostic log and confirm spans are queued but not exported at handler return; 0.75h fix: add force_flush(timeout_millis=10_000) before handler return; add disable_batch=True to openlit.init(); deploy updated Lambda; run 5 manual test invocations and confirm all 5 produce spans in Tempo within 2 seconds of handler return; 0.75h validation and retroactive reconciliation: query OpenAI usage API for May–October 2026 total token counts by day; compare against OpenLIT span token counts for same period; confirm 91.2% gap; compute correct per-document cost estimates; document BatchSpanProcessor–Lambda race condition for team runbook).

Related retainer posts

Track retainer hours without a client portal

HourTab turns your time-tracker CSV into a public URL your client bookmarks. No login. No portal. CSV in, URL out.

See pricing →