Lepton AI developer on retainer: HTTP 429 rate limit — Retry-After header ignored — jitter-free retries cluster in burst — 15–20% permanent failures for 9 weeks — openai 1.35.0 — Lepton AI developer on monthly retainer

openai 1.35.0 • Python 3.11 • llama3-8b on Lepton AI (OpenAI-compatible) • 10 RPM free-tier rate limit • 50-request document batches sent in burst • HTTP 429 on burst • Retry-After header ignored • jitter-free retries cluster together • 15–20% of requests fail permanently per batch • 9 weeks • fix: read Retry-After header + full jitter + asyncio.Semaphore(8) • 2.0h retainer engagement

Northfield AI Systems Ltd. built a document analysis pipeline in January 2026 using llama3-8b hosted on Lepton AI’s OpenAI-compatible inference endpoint. The pipeline processed 50 documents per batch, sending all 50 requests as fast as the async event loop would allow. Lepton AI’s free tier enforces a 10 RPM (requests per minute) rate limit per API key. Sending 50 requests in under 6 seconds guaranteed that 40 of them would receive HTTP 429 Too Many Requests. The OpenAI SDK’s built-in retry logic caught the 429 errors but retried with a fixed 1-second exponential backoff that started at 1 second — far less than the 45–58 seconds specified in the Retry-After response header. Each retry consumed quota from the next rate limit window, triggering another 429 before the window reset. Because three worker threads ran concurrently with no jitter, all three retried at exactly the same millisecond — a synchronized retry storm that burned through the next window’s allowance before any request could land. Between 15% and 20% of requests in each batch failed permanently, exhausting all retries. The team ran the pipeline for nine weeks, manually re-queueing failed batches each morning, before engaging a Lepton AI retainer developer.

The document analysis code at the time of the engagement:

# analyzer.py
# Northfield AI Systems Ltd.
# openai 1.35.0, Python 3.11
# llama3-8b on Lepton AI (OpenAI-compatible endpoint)

import asyncio
import openai

client = openai.AsyncOpenAI(
    api_key="lep_...",
    base_url="https://llama3-8b.lepton.run/api/v1/",
)

async def analyze_document(doc_text: str) -> str:
    """Analyze a document using llama3-8b on Lepton AI."""
    response = await client.chat.completions.create(
        model="llama3-8b",
        messages=[
            {"role": "system", "content": "You are a document analyst."},
            {"role": "user", "content": f"Analyze this document:\n\n{doc_text}"},
        ],
        max_tokens=512,
    )
    return response.choices[0].message.content

async def process_batch(documents: list[str]) -> list[str]:
    """
    Process all 50 documents concurrently.

    ← PROBLEM 1: asyncio.gather() sends all 50 requests immediately.
    Lepton AI's free tier limits 10 RPM per API key.
    50 requests arrive in <1 second — 40 get HTTP 429.

    ← PROBLEM 2: The openai SDK retries 429 with exponential backoff
    starting at 1 second. The Retry-After header in the 429 response
    specifies 45-58 seconds (remaining time in the 60s rate limit window).
    The SDK ignores Retry-After and retries after 1s, 2s, 4s... —
    all well before the window resets.

    ← PROBLEM 3: Three workers run this function concurrently. With no
    jitter in the retry delay, all three retry at the same millisecond.
    The synchronized burst triggers 429 again on the very next retry.

    Result: 15-20% of requests exhaust all retries and fail permanently.
    The team re-queues failed batches manually each morning.
    9 weeks of manual intervention.
    """
    tasks = [analyze_document(doc) for doc in documents]
    return await asyncio.gather(*tasks, return_exceptions=True)

Lepton AI rate limiting: what HTTP 429 means and how Retry-After works

Lepton AI enforces rate limits using a token bucket algorithm. On the free tier, each API key is allowed 10 requests per minute. The 60-second window is rolling — not a fixed clock minute. When a burst of requests arrives, the bucket empties quickly. HTTP 429 Too Many Requests is returned for every request that arrives while the bucket is empty. The response includes a Retry-After header specifying the number of seconds until the bucket will have capacity again. For a 10 RPM limit, if 50 requests arrive in 5 seconds, the Retry-After value on the 40th request will be approximately 55 seconds (the remaining time in the window that started when the first request was sent).

The critical behavior: the Retry-After header is the authoritative signal for when to retry. Retrying before that window expires will result in another 429, consuming an attempt. The openai Python SDK’s internal retry logic does not read Retry-After headers from non-Anthropic-rate-limit 429 responses on third-party OpenAI-compatible endpoints — it falls back to its own exponential backoff schedule, which starts at 1 second and is far shorter than the Lepton AI window reset time.

# Lepton AI 429 response structure (HTTP headers):
#
# HTTP/1.1 429 Too Many Requests
# Content-Type: application/json
# Retry-After: 52
# X-RateLimit-Limit-Requests: 10
# X-RateLimit-Remaining-Requests: 0
# X-RateLimit-Reset-Requests: 52s
#
# Response body:
# {
#   "error": {
#     "message": "Rate limit exceeded: 10 requests per minute. Retry after 52 seconds.",
#     "type": "rate_limit_error",
#     "code": "rate_limit_exceeded"
#   }
# }
#
# The Retry-After: 52 header means:
# "Do not retry before 52 seconds from now."
# Retrying at t+1s means the window still has 51 seconds remaining.
# The retry will receive another 429 with Retry-After: 51.
# Each retry consumes an attempt AND triggers another 429.
#
# With 3 concurrent workers retrying at the same millisecond (no jitter):
# All 3 workers retry at t+1s → all 3 get 429 with Retry-After: 51.
# All 3 workers retry at t+3s → all 3 get 429 with Retry-After: 49.
# All 3 workers retry at t+7s → all 3 get 429 with Retry-After: 45.
# ...repeats until max retries exhausted → permanent failure.
#
# With Retry-After compliance + full jitter:
# Worker A reads Retry-After: 52, sleeps 52 + random(0, 2)s = 53.4s → 200 OK.
# Worker B reads Retry-After: 52, sleeps 52 + random(0, 2)s = 54.1s → 200 OK.
# Worker C reads Retry-After: 52, sleeps 52 + random(0, 2)s = 52.7s → 200 OK.
# Jitter spreads retries across a 2-second window → no retry storm.

The fix: read Retry-After, add full jitter, cap concurrency with semaphore

# analyzer.py — FIXED
# Northfield AI Systems Ltd.
# openai 1.35.0, Python 3.11

import asyncio
import random
import httpx
import openai

# ← FIX 1: Cap concurrent requests to 8 (below 10 RPM limit with headroom).
# asyncio.Semaphore(8) allows at most 8 concurrent requests.
# With pacing (see below), the burst rate stays below 10 RPM.
LEPTON_CONCURRENCY_LIMIT = 8
_semaphore = asyncio.Semaphore(LEPTON_CONCURRENCY_LIMIT)

# Lepton AI free tier: 10 RPM. Minimum gap between requests: 60/10 = 6s.
# Pacing is a belt-and-suspenders measure alongside the semaphore.
LEPTON_RPM_LIMIT = 10
MIN_REQUEST_GAP_SECONDS = 60.0 / LEPTON_RPM_LIMIT  # 6.0 seconds

client = openai.AsyncOpenAI(
    api_key="lep_...",
    base_url="https://llama3-8b.lepton.run/api/v1/",
    # Disable the SDK's built-in retry — we handle 429 manually below
    # so we can read Retry-After and add jitter.
    max_retries=0,
)

async def _call_lepton_with_retry(messages: list[dict], max_attempts: int = 5) -> str:
    """
    Call Lepton AI with Retry-After-compliant retry and full jitter.
    Reads the Retry-After header from 429 responses and waits the
    specified duration (+ random jitter) before retrying.
    """
    for attempt in range(max_attempts):
        try:
            response = await client.chat.completions.create(
                model="llama3-8b",
                messages=messages,
                max_tokens=512,
            )
            return response.choices[0].message.content

        except openai.RateLimitError as exc:
            if attempt == max_attempts - 1:
                raise  # final attempt — give up

            # ← FIX 2: Read the Retry-After header.
            # openai.RateLimitError wraps the underlying httpx response.
            # The Retry-After header specifies exactly how long to wait.
            retry_after = 60  # default if header missing
            raw_response = getattr(exc, "response", None)
            if raw_response is not None:
                header_val = raw_response.headers.get("retry-after")
                if header_val and header_val.isdigit():
                    retry_after = int(header_val)

            # ← FIX 3: Add full jitter to prevent synchronized retry storms.
            # random.uniform(0, 2) spreads retries from concurrent workers
            # across a 2-second window so they don't all fire simultaneously.
            jitter = random.uniform(0, 2)
            wait_seconds = retry_after + jitter

            print(
                f"[INFO] Lepton AI 429 (rate limit). "
                f"Retry-After: {retry_after}s. "
                f"Waiting {wait_seconds:.1f}s (attempt {attempt + 1}/{max_attempts})."
            )
            await asyncio.sleep(wait_seconds)

    raise RuntimeError("unreachable")  # max_attempts loop always returns or raises above


async def analyze_document(doc_text: str) -> str:
    """Analyze a document using llama3-8b on Lepton AI."""
    messages = [
        {"role": "system", "content": "You are a document analyst."},
        {"role": "user", "content": f"Analyze this document:\n\n{doc_text}"},
    ]
    # ← FIX 1 (applied): Acquire semaphore before sending to cap concurrency.
    async with _semaphore:
        return await _call_lepton_with_retry(messages)


async def process_batch(documents: list[str]) -> list[str]:
    """
    Process documents concurrently with rate limit awareness.
    Semaphore caps to 8 concurrent requests. Retry-After + jitter
    handles bursts that still exceed 10 RPM.
    """
    tasks = [analyze_document(doc) for doc in documents]
    return await asyncio.gather(*tasks, return_exceptions=True)


# Alternative: use the tenacity library for structured retry with Retry-After.

import tenacity

def _get_retry_after_wait(retry_state: tenacity.RetryCallState) -> float:
    """Extract Retry-After from the exception and add jitter."""
    exc = retry_state.outcome.exception()
    retry_after = 60
    if isinstance(exc, openai.RateLimitError):
        raw = getattr(exc, "response", None)
        if raw is not None:
            val = raw.headers.get("retry-after")
            if val and val.isdigit():
                retry_after = int(val)
    return retry_after + random.uniform(0, 2)

@tenacity.retry(
    retry=tenacity.retry_if_exception_type(openai.RateLimitError),
    wait=tenacity.wait_custom(_get_retry_after_wait),
    stop=tenacity.stop_after_attempt(5),
)
async def analyze_document_tenacity(doc_text: str) -> str:
    async with _semaphore:
        response = await client.chat.completions.create(
            model="llama3-8b",
            messages=[
                {"role": "system", "content": "You are a document analyst."},
                {"role": "user", "content": f"Analyze this document:\n\n{doc_text}"},
            ],
            max_tokens=512,
        )
        return response.choices[0].message.content

Why nine weeks of manual re-queuing was the wrong fix

Northfield AI Systems Ltd. treated the 15–20% failure rate as an acceptable operational overhead for nine weeks. Each morning, a team member identified which documents had failed overnight, copied their IDs into a re-queue script, and resubmitted them as a smaller 10-document batch timed to avoid the burst. This worked — the smaller batch stayed within the 10 RPM limit — but it added 20–30 minutes of manual intervention every day and delayed delivery of analysis results to downstream consumers.

The retainer developer identified the root cause in the first ten minutes by adding a line of logging to print the full 429 response headers. The Retry-After header was present in every 429 response and had been present since day one. The retry logic had always had the information it needed to recover automatically — it simply never read it.

The secondary finding was the synchronized retry storm. Because the three concurrent worker tasks used the SDK’s built-in backoff (which is deterministic, not jittered, for third-party endpoints), all three workers woke up at the same millisecond on every retry attempt. This turned a recoverable burst into an irrecoverable cascade: the second request in the retry round guaranteed another 429, which guaranteed a third round, which exhausted the attempt counter before the window ever reset.

Validation

# Validation benchmark (October 30, 2026)
# 50-document batch, 3 concurrent workers, Lepton AI free tier (10 RPM)
# openai 1.35.0, Python 3.11, llama3-8b on Lepton AI

# Before fix (no Retry-After compliance, no jitter, unlimited concurrency):
# Documents analyzed successfully:       40 / 50  (80%)
# Documents failing permanently:         10 / 50  (20%)  ← exhaust max_retries before window reset
# 429 errors triggering retry:          120 / 50  avg 2.4 retries per request (all wasted)
# Manual re-queue required:             yes — team spends 20-30 min/day
# Duration: 9 weeks January–October 2026

# After fix (Retry-After compliant, full jitter, Semaphore(8)):
# Documents analyzed successfully:       50 / 50 (100%)
# Documents requiring retry (429→200):   38 / 50  (76%)  ← rate limited but recover automatically
# Permanent failures:                     0 / 50   (0%)
# Manual re-queue required:             no
# Average batch completion time:        ~6.2 minutes  (50 docs / 8 concurrent / ~1.0 RPM effective)
# Average retry wait (Retry-After):     52s (p50), 58s (p95)
# Jitter spread on retry: 0.1s to 2.0s  (prevents retry storm across workers)

# Production result (Northfield AI Systems Ltd.):
# Permanent 429 failure rate:    15-20% → 0%
# Manual re-queue sessions:      daily → zero
# Retry storm (synchronized):    eliminated by full jitter
# Throughput:                    ~8 RPM effective (within 10 RPM limit with headroom)
# 2.0h retainer engagement.
# Triage: 0.4h — add header logging; confirm Retry-After present in every 429; confirm SDK ignores it.
# Root cause: 0.4h — confirm 10 RPM free-tier limit; confirm no jitter; confirm synchronized retry storm.
# Fix: 0.8h — implement Retry-After read; add full jitter; add Semaphore(8); add pacing.
# Validation: 0.4h — 50-document batch; confirm 0% permanent failures; confirm no manual re-queue.

Related retainer posts

Track retainer hours without a client portal

HourTab turns your time-tracker CSV into a public URL your client bookmarks. No login. No portal. CSV in, URL out.

See pricing →