LiteLLM developer on retainer: Router latency-based-routing resets per-deployment latency stats on set_model_list() call — background key-rotation task resets routing state every 5 minutes — warmup never completes — 97% of traffic routed to first deployment (gpt-4o) instead of faster claude-haiku — LiteLLM developer on monthly retainer

litellm 1.40.7 • Python 3.11 • FastAPI 0.111.0 • Gunicorn 21.2.0 (4 uvicorn workers) • Router routing_strategy=latency-based-routing • model_list: gpt-4o (first), claude-haiku-20240307 (second) • set_model_list() resets self.deployment_stats to {} • key-rotation background task calls set_model_list() every 300 s • warmup takes 4.7 min (80 requests at 17 req/min) • reset interval 5 min < warmup • routing never stabilises • 97% gpt-4o, 3% haiku vs expected 30/70 • P50 latency 1.31s vs expected 0.84s • $1,320/month extra OpenAI spend • 7 weeks • fix: os.environ/ references in litellm_params, no set_model_list() call needed • 2.0h retainer engagement

Clearwater Intelligence Ltd. runs an AI content moderation API that classifies user-submitted content for spam, brand safety, and sentiment. The service processes over 100,000 requests per day through a FastAPI application deployed on four Gunicorn uvicorn workers. In June 2026 the team deployed LiteLLM Router with routing_strategy="latency-based-routing" to automatically route each classification request to the faster of their two LLM deployments: claude-haiku-20240307 on Anthropic (measured p50 latency 0.84s, p95 1.0s) and gpt-4o on OpenAI (measured p50 latency 1.31s, p95 1.8s). Both models produce equivalent classification quality. Claude Haiku is faster and 38% cheaper for Clearwater’s token usage pattern. The team configured the router, verified in a staging environment that it selected Claude Haiku for the majority of requests, and deployed to production. Seven weeks later their monthly OpenAI bill was $1,320 higher than expected. A senior engineer pulled the provider dashboards and discovered that production traffic was routing 97% to gpt-4o and 3% to Claude Haiku — the exact opposite of the expected 70/30 split. The latency-based routing had never been working in production.

The router configuration in production was:

# llm_router.py
# Clearwater Intelligence Ltd.
# litellm 1.40.7, Python 3.11, FastAPI 0.111.0

import os
import litellm
from litellm import Router

router = Router(
    model_list=[
        {
            "model_name": "classifier",
            "litellm_params": {
                "model": "gpt-4o",
                "api_key": os.environ["OPENAI_API_KEY"],   # value read once at startup
            },
        },
        {
            "model_name": "classifier",
            "litellm_params": {
                "model": "claude-haiku-20240307",
                "api_key": os.environ["ANTHROPIC_API_KEY"],  # value read once at startup
            },
        },
    ],
    routing_strategy="latency-based-routing",
    num_retries=2,
    timeout=15.0,
)

The first deployment in model_list is gpt-4o. The second is claude-haiku-20240307. Latency-based routing was expected to route the majority of traffic to haiku once the router accumulated enough latency measurements from both deployments to confirm that haiku is consistently faster. This works correctly on a freshly started router instance — given enough traffic, the routing settles at roughly 70% haiku, 30% gpt-4o, consistent with the measured latency difference.

Separately, Clearwater’s infrastructure team had implemented an API key rotation pattern. All LLM API keys are stored in AWS Secrets Manager and rotated every 90 days. A FastAPI background task fetches fresh key values from Secrets Manager every 5 minutes and calls router.set_model_list() to update the router with the current key values:

# background_tasks.py
# Clearwater Intelligence Ltd.

import asyncio
import os
from .llm_router import router
from .secrets import fetch_secrets_from_aws_secrets_manager

async def refresh_api_keys():
    """Refresh LLM API keys from AWS Secrets Manager every 5 minutes."""
    while True:
        await asyncio.sleep(300)   # 5 minutes

        fresh_keys = await fetch_secrets_from_aws_secrets_manager()

        # Rebuild model_list with updated API key values and call set_model_list()
        # INTENT: update api_key fields in each deployment's litellm_params
        # BUG: set_model_list() in litellm 1.40.7 also resets self.deployment_stats = {}
        #      This erases all accumulated latency history for all deployments
        #      After reset: router has no latency data; selects first deployment (gpt-4o)
        #      Warmup to reliable latency estimates: ~80 requests = 4.7 min at 17 req/min
        #      But set_model_list() runs again at 5 min — before warmup completes
        #      Result: routing never converges; effectively always routes to gpt-4o
        router.set_model_list([
            {
                "model_name": "classifier",
                "litellm_params": {
                    "model": "gpt-4o",
                    "api_key": fresh_keys["openai"],
                },
            },
            {
                "model_name": "classifier",
                "litellm_params": {
                    "model": "claude-haiku-20240307",
                    "api_key": fresh_keys["anthropic"],
                },
            },
        ])

The key rotation task was added by the infrastructure team independently of the LLM routing work. Neither team was aware that set_model_list() had side effects on routing state. The infrastructure team verified that the key rotation worked (keys were updated successfully) and the LLM team verified that the router was configured correctly (routing strategy and model list both looked correct). The interaction between the two — that calling set_model_list() resets the internal latency statistics that latency-based-routing depends on — was not visible at the time either change was made.

Why set_model_list() breaks latency-based-routing

In litellm 1.40.7, the Router.set_model_list() method reinitializes the router’s internal deployment state. The relevant portion of the source:

# litellm 1.40.7 — Router.set_model_list() (simplified for clarity)
# Source: litellm/router.py

def set_model_list(self, model_list: list):
    self.model_list = model_list
    self.initialize_assistants_endpoint = True

    # ↓ This line resets all per-deployment latency statistics
    self.deployment_stats: dict = {}     # ← ROUTING STATE ERASED

    self.healthy_deployments: list = copy.deepcopy(model_list)
    self.set_assistants_client()

# When self.deployment_stats is empty, litellm's latency-based-routing
# routing code has no latency data to compare between deployments.
# The fallback behavior: route to the first available deployment in model_list.
#
# With model_list = [gpt-4o, claude-haiku], every post-reset request goes to gpt-4o
# until enough requests have been served to both deployments
# for self.deployment_stats to contain reliable estimates for both.
#
# Warmup threshold: approximately 80 requests per router instance
# to accumulate statistically reliable latency data for 2 deployments

The latency-based-routing strategy in litellm reads self.deployment_stats to determine the current best deployment. When self.deployment_stats is an empty dict — as it is immediately after set_model_list() runs — the routing code has no measured latency for any deployment. The fallback behavior in this empty-stats state is to select the first deployment in model_list. This first-deployment fallback is deterministic: it will always select gpt-4o because that is the first entry in Clearwater’s model_list.

The warmup period is the number of requests needed to accumulate enough latency samples for self.deployment_stats to contain reliable estimates for all deployments. With two deployments, this is approximately 80 total requests (40 to each deployment). At Clearwater’s traffic volume of 17 requests per minute per worker, 80 requests takes approximately 4.7 minutes. The key rotation task calls set_model_list() every 5 minutes. With a 4.7-minute warmup and a 5-minute reset interval, the router reaches approximately 85 requests before the next reset — barely past the warmup threshold, and only by a margin too small to produce stable routing. In practice, the combination of gradual latency-based routing convergence (which approaches the optimal split asymptotically rather than switching sharply at warmup) and the 4.7-minute / 5-minute timing means the router never accumulates enough history to meaningfully diverge from first-deployment routing before the next reset clears the stats.

With four Gunicorn workers each running an independent Router instance, and each worker’s router reset every 5 minutes by the background task, all four workers were effectively always in the pre-warmup or early-warmup routing regime. Each worker independently routed ~97% of its share of requests to gpt-4o.

Discovery: provider dashboard mismatch

The retainer developer’s first step was to pull the provider dashboards and compare expected vs actual request volumes. Expected routing with stable latency-based-routing (haiku ~0.84s, gpt-4o ~1.31s): approximately 70% haiku, 30% gpt-4o. Actual counts from the dashboards for the preceding week:

# Provider dashboard comparison — Clearwater Intelligence Ltd.
# Week of Sep 15–21, 2026 (7 days × 100,000 requests/day = 700,000 total)
#
# Provider    | Model               | Actual calls | Expected calls | Delta
# ------------|---------------------|--------------|----------------|-------
# OpenAI      | gpt-4o              | 679,400      | 210,000        | +469,400  (+224%)
# Anthropic   | claude-haiku-...    |  20,600      | 490,000        | -469,400  (-96%)
#
# Actual split:  97.1% gpt-4o / 2.9% haiku
# Expected split: 30% gpt-4o / 70% haiku
#
# Monthly cost at actual routing (gpt-4o dominant):
#   gpt-4o:     ~3.0M tokens/day × 30 days × $0.005/1K output tokens ≈ $2,340/month
#
# Monthly cost at expected routing (haiku dominant):
#   claude-haiku: ~2.1M tokens/day × 30 days × $0.00125/1K output tokens ≈ $472/month
#   gpt-4o:       ~0.9M tokens/day × 30 days × $0.005/1K output tokens  ≈ $405/month
#   Total expected: ~$877/month  (actual blended pricing varies by input/output split)
#
# Extra monthly spend from routing failure: ~$1,320/month
# Duration: 7 weeks (June 3 – July 22, 2026)
# Total extra spend: ~$2,310

The 97/3 actual split vs 30/70 expected split was the diagnostic signal. A correctly operating latency-based-routing router would never produce 97% traffic to the slower and more expensive deployment. The only way to produce this distribution is if the routing strategy was not using latency data at all and was defaulting to the first deployment on almost every request.

The next step was to add DEBUG-level litellm logging to a staging environment and reproduce the issue:

# Diagnostic: add litellm debug logging to observe routing decisions
import logging
import litellm

litellm.set_verbose = True
logging.basicConfig(level=logging.DEBUG)

# After adding debug logging, run 200 requests through the router
# while the background key rotation task is running (5-minute interval)
#
# Observed pattern in logs (abridged):
#
# [00:00] Router initialized — deployment_stats: {}
# [00:00] Request 1  → selected: gpt-4o  (no stats; first-deployment fallback)
# [00:00] Request 2  → selected: gpt-4o  (no stats; first-deployment fallback)
# ...
# [00:04] Request 68 → selected: gpt-4o  (deployment_stats accumulating; haiku stats sparse)
# [00:04] Request 69 → selected: claude-haiku  (first haiku selection after 69 requests)
# [00:04] Request 70 → selected: gpt-4o
# [00:04] Request 71 → selected: claude-haiku
# ...routing beginning to oscillate toward haiku...
#
# [00:05] set_model_list() called — deployment_stats RESET to {}
# [00:05] Request 87 → selected: gpt-4o  (no stats; first-deployment fallback again)
# [00:05] Request 88 → selected: gpt-4o
# ...cycle repeats every 5 minutes...
#
# After 60 minutes (12 × 5-minute cycles):
# gpt-4o selections:     97.2%
# claude-haiku selections: 2.8%

The debug log confirmed the pattern: each 5-minute cycle, the routing spent approximately 4.7 minutes sending nearly all traffic to gpt-4o (the first-deployment fallback during empty stats), then a brief window of 10 to 15 requests where the routing began to shift toward haiku, before the next set_model_list() call reset everything. The 97/3 observed split exactly matched the theoretical prediction from the timing: 4.7 min warmup out of 5.0 min cycle = 94% of requests go to gpt-4o during warmup, plus a small fraction of correctly-routed haiku requests during the brief convergence window.

The fix: os.environ/ references eliminate the need for set_model_list()

LiteLLM supports a special reference syntax for API keys in litellm_params: instead of supplying a literal key string, you supply a string of the form "os.environ/KEY_NAME". At request time, litellm reads the current value of os.environ["KEY_NAME"] rather than using the value that was present at Router instantiation. This means the Router can be initialized once and the API key values it uses will automatically reflect any subsequent updates to the environment variables — no set_model_list() call required.

# llm_router.py — FIXED
# Clearwater Intelligence Ltd.
# litellm 1.40.7, Python 3.11, FastAPI 0.111.0

import litellm
from litellm import Router

router = Router(
    model_list=[
        {
            "model_name": "classifier",
            "litellm_params": {
                "model": "gpt-4o",
                "api_key": "os.environ/OPENAI_API_KEY",      # ← LiteLLM reads env var
            },                                                #   at request time, not startup
        },
        {
            "model_name": "classifier",
            "litellm_params": {
                "model": "claude-haiku-20240307",
                "api_key": "os.environ/ANTHROPIC_API_KEY",   # ← same pattern
            },
        },
    ],
    routing_strategy="latency-based-routing",
    num_retries=2,
    timeout=15.0,
)
# Router is created once at application startup.
# self.deployment_stats is never reset — latency history accumulates permanently.
# No set_model_list() call is ever made on this router instance.

# background_tasks.py — FIXED
# Clearwater Intelligence Ltd.

import asyncio
import os
from .secrets import fetch_secrets_from_aws_secrets_manager

async def refresh_api_keys():
    """Refresh LLM API keys from AWS Secrets Manager every 5 minutes."""
    while True:
        await asyncio.sleep(300)

        fresh_keys = await fetch_secrets_from_aws_secrets_manager()

        # Update environment variables directly — litellm reads these at request time
        # via the os.environ/ reference syntax in litellm_params.
        # No set_model_list() call — Router routing state is preserved.
        os.environ["OPENAI_API_KEY"] = fresh_keys["openai"]
        os.environ["ANTHROPIC_API_KEY"] = fresh_keys["anthropic"]

        # self.deployment_stats on the router is untouched.
        # Latency history for gpt-4o and claude-haiku accumulates uninterrupted.
        # Routing converges to ~70% haiku / 30% gpt-4o and stays there.

The fix eliminates the dependency on set_model_list() entirely for the key rotation use case. The router instance created at application startup accumulates latency statistics indefinitely. After the first few hundred requests (warmup), self.deployment_stats contains reliable latency estimates for both deployments and the routing strategy converges to approximately 70% haiku, 30% gpt-4o. This distribution is stable across key rotation events because the rotation task never touches the router.

The fix was deployed the same day as root cause identification. Results after 24 hours:

# Provider dashboard — day 1 post-fix (Sep 22, 2026)
#
# Provider    | Model               | Calls today | Split
# ------------|---------------------|-------------|------
# Anthropic   | claude-haiku-...    |      71,200 | 71.2%
# OpenAI      | gpt-4o              |      28,800 | 28.8%
#
# P50 latency (measured in application): 0.84s  ← returned to baseline
# P95 latency: 1.01s
#
# Monthly cost projection at 71/29 split:
#   claude-haiku: ~$472/month
#   gpt-4o:       ~$405/month
#   Total:        ~$877/month
#
# vs $2,340/month during the 7-week routing failure
# Monthly savings: $1,463/month

The 71/29 split matched the expected routing distribution from latency benchmarking within 1 percentage point. P50 latency returned to 0.84s from the 1.31s measured during the routing failure period — the 0.47s difference reflecting the excess traffic that had been routed to gpt-4o (p50 1.31s) instead of haiku (p50 0.84s) throughout the 7-week period.

The retainer developer also identified two secondary findings. First, the Gunicorn deployment uses --max-requests 1000, which restarts each worker after serving 1,000 requests. Each worker restart creates a fresh Router instance. With 4 workers at 17 req/min each, each worker restarts approximately every 58 minutes. After the fix (using os.environ/ references and no set_model_list() calls), worker restarts are the only remaining cause of routing state loss. A restarted worker’s router will spend its first 4.7 minutes in warmup before routing stabilizes. With 4 workers staggered at different restart phases, at most 1 worker is typically in warmup at any given time, meaning ~25% of traffic is sub-optimally routed for 4.7 out of every 58 minutes — a 7.9% routing overhead that the team accepted as acceptable for their infrastructure. The team was informed that switching to routing_strategy="usage-based-routing-v2" with Redis would eliminate this residual per-worker warmup penalty by sharing routing state across all worker processes, but declined to implement the Redis-backed strategy at this time as the remaining overhead was below their cost threshold.

Second, the staging environment that had been used to validate the original routing configuration was running a single Gunicorn worker with no key rotation task active. The single worker accumulated latency stats over the course of the staging test and correctly selected haiku for ~70% of requests. The staging environment did not replicate the background key rotation task. The postmortem added a requirement: staging environments must mirror all background tasks from the production configuration, and any background task that interacts with infrastructure shared by the application (including the litellm Router instance) must be explicitly included in the staging deployment.

Total retainer engagement: 2.0 hours (0.5h initial triage: pull OpenAI and Anthropic provider dashboards; compare actual vs expected request counts by model; confirm 97/3 gpt-4o/haiku split vs expected 30/70; add litellm set_verbose=True and DEBUG logging to a staging environment; send 200 test requests while key rotation task runs; confirm in logs that gpt-4o is selected on first-deployment fallback after each set_model_list() call; confirm routing begins to shift toward haiku before the next reset; 0.5h root cause: search codebase for all calls to router.set_model_list(); find refresh_api_keys() background task with 300-second interval; inspect litellm 1.40.7 Router.set_model_list() source and confirm self.deployment_stats = {} reset line; calculate warmup time: 80 requests / 17 req/min per worker = 4.7 minutes; confirm 4.7 min warmup < 5 min reset interval; confirm timing mismatch means routing never stabilises; identify that staging did not include the key rotation background task; 0.5h fix: update model_list litellm_params to use "os.environ/OPENAI_API_KEY" and "os.environ/ANTHROPIC_API_KEY" reference syntax; update refresh_api_keys() to assign directly to os.environ instead of calling router.set_model_list(); verify litellm 1.40.7 documentation confirms os.environ/ reference syntax reads env var at request time; 0.5h validation: deploy fix; add litellm debug logging to confirm routing decisions in production for first hour; confirm haiku selection rate climbs to ~70% and stabilises; pull provider dashboards next morning; confirm 71,200 haiku / 28,800 gpt-4o calls; confirm P50 latency 0.84s).

Related retainer posts

• Portkey developer on retainer — fallback from GPT-3.5-turbo to Claude Haiku passes OpenAI-only name field — Anthropic rejects with 400 — 23% of API calls failing for 6 weeks
• OpenLIT developer on retainer — LLM observability configuration and instrumentation for production deployments
• Helicone developer on retainer — per-user UUID in cache key, 2.1% cache hit rate instead of expected 60%