Groq developer on retainer: streaming chat pauses for 3–4 seconds mid-response when prompt exceeds 2,048 tokens — groq 0.9.0 TPM budget exhausted mid-stream — connection appears hung with no error — 12% of production requests affected — llama-3.1-70b-versatile — Groq developer on monthly retainer

groq 0.9.0 • Python 3.11 • llama-3.1-70b-versatile • stream=True • p95=180ms for short prompts • p95=4,200ms for prompts over 2,048 tokens • TPM budget exhausted mid-stream • stream pauses silently 3–4s • no HTTP error • 12% of production requests • fix: tiktoken pre-count + map-reduce split • 2.5h retainer engagement

Westbrook Inference Ltd. built a production streaming chat endpoint in October 2026 using Groq’s API for low-latency LLM responses. Their benchmark showed a consistent p95 latency of 180ms — exactly the Groq advantage they had signed up for. Production monitoring told a different story: 12% of requests showed p95 latency of 4,200ms — a 23× degradation. Those slow requests had one thing in common: input prompts exceeding 2,048 tokens. They engaged a Groq retainer developer to investigate.

The streaming endpoint at the time of the engagement:

# chat_endpoint.py
# Westbrook Inference Ltd.
# groq 0.9.0, Python 3.11

from fastapi import FastAPI
from fastapi.responses import StreamingResponse
from groq import Groq

app = FastAPI()
groq_client = Groq(api_key="GROQ_API_KEY")

@app.post("/chat")
async def chat(messages: list[dict], system_prompt: str = ""):
    """
    Stream chat completions via Groq.
    Typical p95 latency: 180ms for short prompts.
    Problem: 12% of requests show p95=4,200ms.
    All slow requests have input token count > 2,048.
    """
    def stream_response():
        stream = groq_client.chat.completions.create(
            model="llama-3.1-70b-versatile",
            messages=[
                {"role": "system", "content": system_prompt},
                *messages
            ],
            stream=True,
            # ← PROBLEM: no pre-flight token count
            # No check against TPM budget before API call
            # No stream read timeout
            # No X-RateLimit-Remaining-Tokens monitoring
        )
        for chunk in stream:
            # ← When TPM budget is exhausted mid-stream:
            # This for-loop blocks here. No chunk arrives.
            # No exception is raised. No HTTP error is returned.
            # The Groq API holds the connection open, streaming nothing,
            # until the TPM budget refills (~3-4 seconds).
            # After refill, remaining chunks arrive normally.
            # Total latency: 180ms normal + 3,400ms pause + ~620ms resumption
            content = chunk.choices[0].delta.content
            if content:
                yield content

    return StreamingResponse(stream_response(), media_type="text/plain")

How TPM budget exhaustion causes silent streaming pauses

Groq enforces a per-account token-per-minute (TPM) limit. The limit is enforced on a rolling 60-second window. Every API request consumes from this budget: input tokens (from the prompt) are consumed when the request is received; output tokens (generated by the model) are consumed as they are generated during streaming.

When the TPM budget is exhausted, Groq’s behavior for streaming requests differs from its behavior for non-streaming requests. For non-streaming requests (stream=False), Groq returns HTTP 429 Too Many Requests immediately, and the groq Python client raises groq.RateLimitError. For streaming requests (stream=True), Groq does not close the connection on TPM exhaustion. Instead, it pauses the stream — the HTTP connection remains open, but no more server-sent event chunks arrive. The client’s for-loop blocks at the chunk iteration point. No exception is raised. No timeout fires (unless explicitly configured). The stream resumes after 3–4 seconds when the 60-second window refills enough budget to continue.

# Diagnosing TPM exhaustion via response headers
# groq 0.9.0, Python 3.11

import time
from groq import Groq

client = Groq(api_key="GROQ_API_KEY")

def chat_with_header_logging(prompt: str) -> str:
    """Send a chat request and log Groq rate limit headers."""
    start = time.monotonic()
    chunks = []

    stream = client.chat.completions.create(
        model="llama-3.1-70b-versatile",
        messages=[{"role": "user", "content": prompt}],
        stream=True,
    )

    # Access response headers from the stream object
    # groq 0.9.0: headers available on stream.response.headers
    remaining_tokens = stream.response.headers.get("x-ratelimit-remaining-tokens")
    reset_tokens = stream.response.headers.get("x-ratelimit-reset-tokens")
    limit_tokens = stream.response.headers.get("x-ratelimit-limit-tokens")

    print(f"TPM limit: {limit_tokens}")
    print(f"TPM remaining at request start: {remaining_tokens}")
    print(f"TPM refill in: {reset_tokens}")

    prev_chunk_time = start
    for chunk in stream:
        now = time.monotonic()
        inter_chunk_gap = now - prev_chunk_time
        if inter_chunk_gap > 1.0:
            # Gap > 1 second between chunks = TPM pause in progress
            print(f"  ⚠ Inter-chunk gap: {inter_chunk_gap:.2f}s (TPM pause suspected)")
        content = chunk.choices[0].delta.content
        if content:
            chunks.append(content)
        prev_chunk_time = now

    total_latency = time.monotonic() - start
    print(f"Total latency: {total_latency:.3f}s")
    return "".join(chunks)

# Output for a short prompt (800 tokens):
# TPM limit: 6000
# TPM remaining at request start: 4,821
# TPM refill in: 18s
# Total latency: 0.187s
# ← Normal: plenty of TPM budget, no pause.

# Output for a long prompt (2,600 tokens) during peak traffic:
# TPM limit: 6000
# TPM remaining at request start: 1,240
# TPM refill in: 42s
# ← Warning: only 1,240 tokens remaining, request needs 2,600 input + ~400 output = 3,000 total
#   TPM budget will be exhausted mid-generation after ~1,240 tokens of output.
#   Inter-chunk gap: 3.42s (TPM pause suspected)   ← pause detected during generation
# Total latency: 4.218s
# ← Confirmed: TPM budget exhausted mid-stream; 3.42s pause; stream resumed after budget refill.

Measuring the impact: latency distribution by prompt token count

# Latency analysis from 1,000 production requests (7-day sample)
# groq 0.9.0, Python 3.11, llama-3.1-70b-versatile

# Segmented by input token count (pre-calculated post-hoc via tiktoken)
import tiktoken
import statistics

enc = tiktoken.encoding_for_model("gpt-4")  # approximate Llama tokenizer

# Request log: (input_text, response_time_seconds)
short_latencies = []   # input tokens < 1,000
medium_latencies = []  # input tokens 1,000-2,048
long_latencies = []    # input tokens > 2,048

for input_text, response_time in request_log:
    token_count = len(enc.encode(input_text))
    if token_count < 1000:
        short_latencies.append(response_time)
    elif token_count <= 2048:
        medium_latencies.append(response_time)
    else:
        long_latencies.append(response_time)

def p95(latencies):
    return sorted(latencies)[int(len(latencies) * 0.95)]

print("Latency by prompt token bucket:")
print(f"Short (<1,000 tokens)  n={len(short_latencies)}: p50={statistics.median(short_latencies):.0f}ms  p95={p95(short_latencies)*1000:.0f}ms")
print(f"Medium (1k-2k tokens)  n={len(medium_latencies)}: p50={statistics.median(medium_latencies):.0f}ms  p95={p95(medium_latencies)*1000:.0f}ms")
print(f"Long (>2,048 tokens)   n={len(long_latencies)}: p50={statistics.median(long_latencies):.0f}ms  p95={p95(long_latencies)*1000:.0f}ms")

# Output:
# Latency by prompt token bucket:
# Short (<1,000 tokens)  n=613: p50=142ms  p95=198ms   ← Excellent Groq latency
# Medium (1k-2k tokens)  n=267: p50=156ms  p95=312ms   ← Still within SLA
# Long (>2,048 tokens)   n=120: p50=2,840ms  p95=4,218ms  ← Severely degraded

# ← The bimodal distribution is clear:
# Short and medium prompts show Groq's expected sub-200ms performance.
# Long prompts (12% of requests) show 4,200ms p95 — 23x worse than benchmark.
# The 120 long-prompt requests map to the 12% slow requests reported in monitoring.

# TPM consumption analysis:
# llama-3.1-70b-versatile TPM limit (free/developer plan): 6,000 tokens/minute
# Average long prompt: 2,800 input tokens + 380 output = 3,180 tokens/request
# At 50 RPM peak: 50 * (proportion of long prompts * 3,180 + short tokens)
# Long prompts alone: 120/1000 * 50 req/min * 3,180 = 19,080 tokens/min potential
# TPM limit 6,000 is exceeded when multiple long-prompt requests arrive concurrently
# At 50 RPM: ~2 long-prompt requests per minute average; burst of 3+ exhausts 6,000 TPM

The fix: pre-count tokens and split long prompts before sending to Groq

# chat_endpoint.py — FIXED
# Westbrook Inference Ltd.
# groq 0.9.0, Python 3.11

import tiktoken
import time
from fastapi import FastAPI
from fastapi.responses import StreamingResponse
from groq import Groq

app = FastAPI()
groq_client = Groq(api_key="GROQ_API_KEY")
enc = tiktoken.encoding_for_model("gpt-4")  # approximate Llama tokenizer

MAX_DIRECT_TOKENS = 1_800   # threshold for direct API call vs map-reduce split
                             # 1,800 leaves headroom for output tokens within 6,000 TPM limit

def count_tokens(messages: list[dict], system_prompt: str) -> int:
    total = 0
    if system_prompt:
        total += len(enc.encode(system_prompt))
    for msg in messages:
        total += len(enc.encode(msg.get("content", "")))
    return total

def stream_direct(messages: list[dict], system_prompt: str):
    """Stream a single API call when prompt is within TPM-safe threshold."""
    stream = groq_client.chat.completions.create(
        model="llama-3.1-70b-versatile",
        messages=[
            {"role": "system", "content": system_prompt},
            *messages
        ],
        stream=True,
        timeout=10.0,   # ← FIX: stream read timeout
                        # Raises ReadTimeout if no chunk arrives for 10 seconds.
                        # Surfaces TPM-induced pauses as measurable errors
                        # rather than silent hangs. 10s > typical 3-4s pause,
                        # but catches runaway pauses from sustained TPM exhaustion.
    )
    for chunk in stream:
        content = chunk.choices[0].delta.content
        if content:
            yield content

def stream_with_map_reduce(messages: list[dict], system_prompt: str):
    """Split long prompts into chunks, process separately, yield merged result."""
    # Collect the user messages as a single document for splitting
    user_content = " ".join(
        msg["content"] for msg in messages if msg["role"] == "user"
    )
    tokens = enc.encode(user_content)

    # Split into ~1,000-token chunks with 100-token overlap
    chunk_size = 1000
    overlap = 100
    chunks = []
    for i in range(0, len(tokens), chunk_size - overlap):
        chunk_tokens = tokens[i:i + chunk_size]
        chunks.append(enc.decode(chunk_tokens))

    # Map: process each chunk independently
    partial_results = []
    for chunk_text in chunks:
        response = groq_client.chat.completions.create(
            model="llama-3.1-70b-versatile",
            messages=[
                {"role": "system", "content": system_prompt},
                {"role": "user", "content": chunk_text}
            ],
            stream=False,    # Non-streaming for map phase (simpler, avoids timeout complexity)
            max_tokens=800,  # Each chunk produces a partial result
            timeout=15.0,
        )
        partial_results.append(response.choices[0].message.content)

    # Reduce: synthesize partial results into final answer
    synthesis_prompt = (
        "Synthesize these partial analysis results into one coherent final answer:\n\n"
        + "\n---\n".join(f"Part {i+1}:\n{r}" for i, r in enumerate(partial_results))
    )
    stream = groq_client.chat.completions.create(
        model="llama-3.1-70b-versatile",
        messages=[
            {"role": "system", "content": system_prompt},
            {"role": "user", "content": synthesis_prompt}
        ],
        stream=True,
        timeout=10.0,
    )
    for chunk in stream:
        content = chunk.choices[0].delta.content
        if content:
            yield content

@app.post("/chat")
async def chat(messages: list[dict], system_prompt: str = ""):
    input_tokens = count_tokens(messages, system_prompt)

    if input_tokens <= MAX_DIRECT_TOKENS:
        generator = stream_direct(messages, system_prompt)
    else:
        # Long prompt: use map-reduce split to stay within TPM budget
        generator = stream_with_map_reduce(messages, system_prompt)

    return StreamingResponse(generator, media_type="text/plain")

Validation

# Load test validation (October 28 2026)
# 1,000 requests simulating production traffic mix: 88% short, 12% long prompts

# Before fix:
# Short prompts  p50=142ms  p95=198ms   (n=880)
# Long prompts   p50=2,840ms  p95=4,218ms  (n=120) ← TPM pause
# Overall p95:   3,890ms

# After fix (map-reduce for long prompts):
# Short prompts (direct)  p50=144ms  p95=201ms   (n=880)  ← unchanged
# Long prompts (map-reduce) p50=1,240ms  p95=1,890ms  (n=120)  ← no TPM pauses
# Overall p95:   890ms  ← within SLA (<1,000ms agreed)

# Accuracy impact of map-reduce (quality evaluation):
# 50 long-prompt test cases evaluated by domain experts
# Direct (pre-split, no TPM pause): accuracy baseline 100%
# Map-reduce (2-3 chunks + synthesis): accuracy 91.4%
# Accuracy loss acceptable for Westbrook's summarization use case
# For use cases requiring higher accuracy: increase chunk overlap or use Llama-3.1-405B for synthesis

# Rate limit header monitoring (added to all requests):
# Groq response headers: x-ratelimit-remaining-tokens, x-ratelimit-reset-tokens
# Alert rule: if remaining_tokens < 1,000 → send alert to ops channel
# Alert rule: if inter-chunk gap > 1.5s → log as TPM_PAUSE event, increment counter
# Alert rule: if TPM_PAUSE events > 10/minute → escalate to oncall
# After fix: 0 TPM_PAUSE events at peak load (map-reduce prevents budget exhaustion)

Related retainer posts

Track retainer hours without a client portal

HourTab turns your time-tracker CSV into a public URL your client bookmarks. No login. No portal. CSV in, URL out.

See pricing →