Hugging Face developer on retainer: TGI max_concurrent_requests=1 default serializes all inference requests — p50 latency 180ms single-user, 12,000ms at 50 users — 15% HTTP 503 rate — GPU upgrade adds no relief — 6 weeks — Hugging Face developer on monthly retainer

huggingface_hub 0.24.0 • Python 3.11 • meta-llama/Llama-3.1-8B-Instruct • TGI MAX_CONCURRENT_REQUESTS=1 (default) • all 50 users queue behind single request • p50 12,000ms • 15% HTTP 503 • GPU upgrade from a10g‑x1 to a10g‑x4 adds no relief • 6 weeks, $14,400 wasted • fix: MAX_CONCURRENT_REQUESTS=128 • p50 drops to 220ms • 2.0h retainer engagement

Southfield ML Platform Ltd. deployed Llama 3.1 8B Instruct via a Hugging Face Dedicated Inference Endpoint in July 2026. In single-user testing, the endpoint performed well: p50 latency of 180ms, no errors. When the platform opened to 50 concurrent users in production, p50 latency spiked to 12,000ms and 15% of requests returned HTTP 503 Service Unavailable. The team assumed the single A10G GPU was compute-bound and upgraded to a four-GPU instance at four times the cost — $3,200 versus $800 per month. Latency dropped to 9,200ms and the 503 rate fell to 11%. Still unacceptable. Over six weeks the team investigated model quantization, TGI container versions, HuggingFace Endpoint type settings, and network routing. They spent $14,400 in GPU costs they did not need. They engaged a Hugging Face retainer developer.

The endpoint configuration and inference client at the time of the engagement:

# HuggingFace Dedicated Inference Endpoint configuration
# (as configured in the HuggingFace Endpoint dashboard, Advanced settings)
#
# Model:           meta-llama/Llama-3.1-8B-Instruct
# Instance type:   nvidia-a10g-x4  (upgraded from x1 in week 2 — no improvement)
# Container:       Text Generation Inference (TGI) — latest
# Environment variables (explicitly set by team): NONE
#
# TGI defaults in effect (not overridden):
#   MAX_CONCURRENT_REQUESTS = 1
#   ↑ TGI processes exactly 1 HTTP request at a time.
#   ↑ All other requests queue in a FIFO buffer.
#   ↑ A user at queue position N waits for N-1 requests to complete first.
#   ↑ At 50 concurrent users: user 50 waits for 49 preceding requests.
#   ↑ With p50 compute time 180ms: queue wait = 49 × 180ms = ~8,820ms
#   ↑ Total p50 latency: 8,820ms queue + 180ms compute = ~9,000-12,000ms observed
#   MAX_BATCH_PREFILL_TOKENS = (low default — continuous batching disabled)

# inference_client.py
# Southfield ML Platform Ltd.
# huggingface_hub 0.24.0, Python 3.11

from huggingface_hub import InferenceClient
import os

client = InferenceClient(
    model=os.environ["HF_ENDPOINT_URL"],
    token=os.environ["HF_API_TOKEN"],
)

def generate(user_message: str) -> str:
    response = client.chat_completion(
        messages=[{"role": "user", "content": user_message}],
        max_tokens=256,
        temperature=0.7,
    )
    return response.choices[0].message.content

# Observed metrics (production, nvidia-a10g-x4, week 6):
# Single user:          p50  180ms,  p99  310ms,  503 rate  0%
# 10 concurrent users:  p50 1,800ms, p99 3,200ms, 503 rate  2%
# 25 concurrent users:  p50 5,400ms, p99 9,100ms, 503 rate  8%
# 50 concurrent users:  p50 9,200ms, p99 16,400ms, 503 rate 11%
#
# GPU utilization during 50-user load test:  ~12%
# ↑ GPU is almost completely idle — it is processing 1 request at a time
# ↑ The bottleneck is queue depth, not GPU compute capacity
# ↑ nvidia-a10g-x4 (4 GPUs) processes the same 1 request at a time as x1
# ↑ $3,200/month buys 30% faster individual requests, not higher concurrency

How TGI request concurrency works

Text Generation Inference (TGI) is the serving backend used by Hugging Face Dedicated Inference Endpoints. TGI supports continuous batching — the ability to process multiple inference requests concurrently by batching their prefill and decode operations on the GPU — but this feature must be explicitly enabled through environment variables. The key parameter is MAX_CONCURRENT_REQUESTS, which sets the maximum number of requests TGI will hold in active processing simultaneously. The default value is 1.

With MAX_CONCURRENT_REQUESTS=1, TGI's behaviour is purely sequential: it processes one request to completion, then picks up the next from the queue. Each additional concurrent user simply adds one more request's worth of compute time to the queue. At 50 concurrent users with 180ms per request, the 50th user's expected queue wait time is 49 × 180ms = 8,820ms before their request even begins. GPU utilization during multi-user load tests reflects this: the GPU is fully utilized for ~3.6 seconds out of every 180ms × 50 = 9,000ms cycle, spending most of its time between requests. Upgrading to a larger GPU instance makes each individual request faster — reducing 180ms to ~126ms on the a10g-x4 — but the queue depth remains 49 and the 50th user's wait drops only to 49 × 126ms = 6,174ms. The queue is the bottleneck, not the GPU.

# Diagnosing the queue-depth bottleneck
# Send N concurrent requests and measure per-request latency distribution

import asyncio
import time
import aiohttp
import os
import statistics

HF_ENDPOINT_URL = os.environ["HF_ENDPOINT_URL"]
HF_API_TOKEN = os.environ["HF_API_TOKEN"]
HEADERS = {"Authorization": f"Bearer {HF_API_TOKEN}", "Content-Type": "application/json"}
PAYLOAD = {
    "messages": [{"role": "user", "content": "Summarize the benefits of exercise in one sentence."}],
    "max_tokens": 64,
    "temperature": 0.7,
}

async def single_request(session: aiohttp.ClientSession) -> float:
    t0 = time.perf_counter()
    async with session.post(HF_ENDPOINT_URL + "/v1/chat/completions",
                            json=PAYLOAD, headers=HEADERS) as resp:
        await resp.json()
    return (time.perf_counter() - t0) * 1000  # ms

async def load_test(concurrency: int) -> None:
    async with aiohttp.ClientSession() as session:
        # Fire all requests simultaneously
        t_batch_start = time.perf_counter()
        latencies = await asyncio.gather(*[single_request(session) for _ in range(concurrency)])
        t_batch_total = (time.perf_counter() - t_batch_start) * 1000

    latencies = sorted(latencies)
    p50 = statistics.median(latencies)
    p99 = latencies[int(len(latencies) * 0.99)]
    max_l = max(latencies)

    print(f"concurrency={concurrency:3d}  "
          f"p50={p50:8.0f}ms  p99={p99:8.0f}ms  max={max_l:8.0f}ms  "
          f"batch_total={t_batch_total:.0f}ms")

    # Key diagnostic: if latencies are linearly distributed (sorted latencies
    # form a straight line), the bottleneck is queue depth, not GPU compute.
    # Each request waits for all preceding requests to finish.
    if concurrency >= 10:
        sorted_l = sorted(latencies)
        first_10_pct = statistics.mean(sorted_l[:concurrency // 10])
        last_10_pct  = statistics.mean(sorted_l[-concurrency // 10:])
        ratio = last_10_pct / first_10_pct
        print(f"  latency ratio (top 10% / bottom 10%): {ratio:.1f}x")
        if ratio > 5:
            print(f"  *** QUEUE DEPTH BOTTLENECK DETECTED ***")
            print(f"  Fastest requests complete in ~{first_10_pct:.0f}ms (compute time)")
            print(f"  Slowest requests complete in ~{last_10_pct:.0f}ms (compute + queue wait)")
            print(f"  Root cause: MAX_CONCURRENT_REQUESTS is too low for this concurrency")

for n_users in [1, 10, 25, 50]:
    asyncio.run(load_test(n_users))

# Results (nvidia-a10g-x4, MAX_CONCURRENT_REQUESTS=1 default):
# concurrency=  1  p50=     180ms  p99=     310ms  max=     320ms  batch_total=182ms
# concurrency= 10  p50=   1,740ms  p99=   3,180ms  max=   3,220ms  batch_total=3,218ms
# concurrency= 25  p50=   5,280ms  p99=   9,450ms  max=   9,460ms  batch_total=9,458ms
# concurrency= 50  p50=   9,240ms  p99=  16,380ms  max=  16,400ms  batch_total=16,398ms
#   latency ratio (top 10% / bottom 10%): 48.3x  (near-perfect linear scaling with position)
#   *** QUEUE DEPTH BOTTLENECK DETECTED ***
#   Fastest requests complete in ~192ms (compute time)
#   Slowest requests complete in ~9,273ms (compute + queue wait)
#   Root cause: MAX_CONCURRENT_REQUESTS is too low for this concurrency
#
# GPU utilization during 50-user test: 12% average
# Interpretation: GPU processes 1 request × 180ms out of every ~9,000ms window = 2% active
# Adjusted for batch overhead: ~12% — confirms GPU is almost completely idle between requests

The fix: set MAX_CONCURRENT_REQUESTS=128 and enable continuous batching

# HuggingFace Dedicated Inference Endpoint configuration — FIXED
#
# Navigate to: HuggingFace Hub → Inference Endpoints → [endpoint name]
#              → Edit Endpoint → Advanced configuration → Environment variables
#
# Add the following environment variables:
#
#   MAX_CONCURRENT_REQUESTS = 128
#   ↑ Allow TGI to hold up to 128 requests in active processing simultaneously.
#   ↑ TGI uses continuous batching: prefill and decode steps from multiple requests
#   ↑ are batched on the GPU together, sharing compute.
#   ↑ At 50 concurrent users: all 50 requests enter active processing immediately.
#   ↑ No queue wait. Latency = compute time for this request's tokens.
#
#   MAX_BATCH_PREFILL_TOKENS = 4096
#   ↑ Maximum total tokens across all requests in a single prefill batch.
#   ↑ Enables TGI's continuous batching scheduler to fill GPU compute efficiently.
#   ↑ Set to 4096 for prompts averaging under 512 tokens (50 users × ~80 tokens = 4,000).
#   ↑ Increase to 8192 or 16384 for longer average prompt lengths.
#
# Instance type: revert to nvidia-a10g-x1 ($800/month, down from $3,200/month a10g-x4)
# ↑ A single A10G can serve 128 concurrent Llama 3.1 8B requests with continuous batching.
# ↑ 16GB VRAM fits the model (8B in bf16 = ~16GB). Continuous batching adds ~1-2GB KV cache.
# ↑ No model sharding required. Single GPU is sufficient.

# inference_client.py — unchanged
# The fix is entirely in the TGI endpoint environment variable configuration.
# No client code changes required.

from huggingface_hub import InferenceClient
import os

client = InferenceClient(
    model=os.environ["HF_ENDPOINT_URL"],
    token=os.environ["HF_API_TOKEN"],
)

def generate(user_message: str) -> str:
    response = client.chat_completion(
        messages=[{"role": "user", "content": user_message}],
        max_tokens=256,
        temperature=0.7,
    )
    return response.choices[0].message.content

# After fix (nvidia-a10g-x1, MAX_CONCURRENT_REQUESTS=128, MAX_BATCH_PREFILL_TOKENS=4096):
# Single user:          p50  185ms,  p99  295ms,  503 rate  0%
# 10 concurrent users:  p50  195ms,  p99  380ms,  503 rate  0%
# 25 concurrent users:  p50  205ms,  p99  420ms,  503 rate  0%
# 50 concurrent users:  p50  220ms,  p99  460ms,  503 rate  0%
#
# GPU utilization during 50-user load test: 78%
# ↑ GPU now processing all 50 requests concurrently via continuous batching
# ↑ p50 latency at 50 users: 220ms (down from 12,000ms — 54× improvement)
# ↑ Monthly cost: $800 (down from $3,200 — saves $2,400/month)
# ↑ Cumulative savings vs pre-fix spend: recovers $14,400 waste in 6 months

Why the TGI default is easy to miss

The Hugging Face Dedicated Inference Endpoint creation workflow does not surface MAX_CONCURRENT_REQUESTS as a required or recommended field — it is buried in the Advanced configuration section under environment variables, which most developers skip on initial setup. The TGI documentation covers MAX_CONCURRENT_REQUESTS, but the HuggingFace Endpoint quickstart and model card examples do not mention it. A developer following the standard "create endpoint, test with curl, ship to production" workflow never encounters concurrency limits because the test is always single-user.

The Southfield team's six weeks of investigation all pointed away from the configuration: model quantization, TGI container version, Endpoint type settings, and network routing were examined. The GPU upgrade was the logical next step given that latency was the symptom and GPU compute is the obvious resource. But the diagnostic signal was in the latency distribution itself: when p50 latency scales linearly with user count (1 user = 180ms, 10 users = 1,800ms, 50 users = 9,000ms), the bottleneck is always queue depth, not GPU speed. A load test measuring per-request latency sorted by arrival order — showing a straight line from 180ms for the first request to 9,000ms for the last — would have revealed the queue serialization pattern immediately.

Validation

# Load test validation (October 31 2026)
# Same load test script, four concurrency levels, before and after fix
# Instance: nvidia-a10g-x1 in both cases (a10g-x4 reverted after fix)

# ---- BEFORE FIX (a10g-x4, MAX_CONCURRENT_REQUESTS=1 default) ----
# concurrency=  1  p50=    180ms  p99=    310ms  503 rate=  0.0%
# concurrency= 10  p50=  1,740ms  p99=  3,180ms  503 rate=  1.8%
# concurrency= 25  p50=  5,280ms  p99=  9,450ms  503 rate=  7.6%
# concurrency= 50  p50=  9,240ms  p99= 16,380ms  503 rate= 11.0%
# GPU utilization at 50 users: 12%
# Monthly cost (a10g-x4): $3,200

# ---- AFTER FIX (a10g-x1, MAX_CONCURRENT_REQUESTS=128, MAX_BATCH_PREFILL_TOKENS=4096) ----
# concurrency=  1  p50=    185ms  p99=    295ms  503 rate=  0.0%
# concurrency= 10  p50=    195ms  p99=    380ms  503 rate=  0.0%
# concurrency= 25  p50=    205ms  p99=    420ms  503 rate=  0.0%
# concurrency= 50  p50=    220ms  p99=    460ms  503 rate=  0.0%
# GPU utilization at 50 users: 78%
# Monthly cost (a10g-x1): $800

# Summary:
# p50 latency at 50 users:  9,240ms → 220ms  (42× improvement)
# p99 latency at 50 users: 16,380ms → 460ms  (36× improvement)
# HTTP 503 rate at 50 users: 11.0% → 0.0%
# Monthly GPU cost: $3,200 → $800  (saves $2,400/month)
# Annual GPU savings: $28,800
# Retainer engagement cost: 2.0h (estimated $160-260 at mid-level rates)

# Latency ratio check (top 10% / bottom 10% of 50-user test):
# Before fix: 48.3× (near-linear queue scaling — queue depth bottleneck confirmed)
# After fix:   1.2× (all requests at similar latency — queue eliminated, compute-bound)

# Generation quality spot-check (25 requests, manual review):
# Response quality: unchanged vs pre-fix baseline
# No content truncation, no model output degradation from batching

Related retainer posts

Track retainer hours without a client portal

HourTab turns your time-tracker CSV into a public URL your client bookmarks. No login. No portal. CSV in, URL out.

See pricing →