Baseten developer on retainer: HTTP 503 during scale-from-zero cold-start with no retry — 2–3% morning failures for 8 weeks — requests 2.31.0 — Baseten developer on monthly retainer

requests 2.31.0 • Python 3.11 • custom Truss model on Baseten • min_scale=0 deployment • Baseten cold-start 30–60s when scaling from zero replicas • HTTP 503 during scale-up treated as permanent error • no retry logic • 2–3% morning error rate • 8 weeks • fix: exponential backoff retry on 503 (2s/4s/8s, max 3 retries) • 2.0h retainer engagement

Eastfield MLOps Ltd. deployed a custom document classification model on Baseten in February 2026 for a contract analysis pipeline processing 500 documents per day. The deployment was configured with min_scale=0 to eliminate GPU billing during overnight and weekend idle periods. Each morning, when the first batch of documents arrived from client uploads, the Baseten deployment had scaled to zero replicas overnight. Baseten required 30 to 60 seconds to provision a new GPU replica, load the model weights, and start the Truss server. During this cold-start window, every incoming request received HTTP 503 Service Unavailable. The Python service calling Baseten treated HTTP 503 as a permanent failure and surfaced an error to the requesting client immediately. No retry logic existed anywhere in the prediction pipeline. Between 2% and 3% of daily document classifications failed every morning for eight weeks. The team filed multiple Baseten support tickets reporting “intermittent 503 errors” and investigated their Truss model packaging, Docker image build, and network configuration before engaging a Baseten retainer developer.

The document classification service code at the time of the engagement:

# classifier.py
# Eastfield MLOps Ltd.
# requests 2.31.0, Python 3.11
# Custom Truss model on Baseten (contract classification)

import requests
import os

BASETEN_API_KEY = os.environ["BASETEN_API_KEY"]
BASETEN_MODEL_ID = "abcd1234"  # Baseten model deployment ID
BASETEN_URL = f"https://model-{BASETEN_MODEL_ID}.api.baseten.co/production/predict"

def classify_document(document_text: str) -> dict:
    """
    Classify a contract document using the Baseten-deployed model.
    Returns a dict with 'category', 'confidence', and 'clauses' fields.
    """
    response = requests.post(
        BASETEN_URL,
        headers={
            "Authorization": f"Api-Key {BASETEN_API_KEY}",
            "Content-Type": "application/json",
        },
        json={"text": document_text},
        timeout=30,
    )

    # ← PROBLEM: raise_for_status() raises HTTPError immediately on 503.
    #
    # Baseten returns HTTP 503 Service Unavailable during the cold-start window
    # when the deployment is scaling from 0 replicas. The 503 is transient —
    # it lasts for the 30-60 seconds required to provision a new GPU instance,
    # load model weights, and start the Truss server.
    #
    # After the cold-start completes, the same request would succeed with HTTP 200.
    # But raise_for_status() on the first 503 raises HTTPError immediately:
    #   requests.exceptions.HTTPError: 503 Server Error: Service Unavailable
    #
    # This exception propagates to the caller, which marks the document as
    # "classification failed" and returns an error to the upload pipeline.
    # The document is not retried. The error is permanent from the user's perspective.
    #
    # 2-3% of documents are uploaded during the 30-60s morning cold-start window
    # and fail every day. The pattern: 503 errors cluster between 8:00-8:02 AM
    # (first uploads after overnight idle), then disappear once the replica is warm.
    response.raise_for_status()

    return response.json()

Baseten cold-start: what HTTP 503 means and when it occurs

Baseten deployments with min_scale=0 save GPU costs by scaling to zero replicas after a configurable inactivity timeout (typically 15–30 minutes). When a request arrives at a scaled-down deployment, Baseten’s autoscaler initiates a cold-start sequence: provision a GPU instance, pull the model Docker image, allocate VRAM, load model weights, and start the Truss prediction server. During this 30–60 second window, Baseten’s load balancer returns HTTP 503 for every incoming request.

The HTTP 503 during cold-start is not an error in the traditional sense. It is a transient unavailability that resolves automatically once the replica is healthy. The correct client behavior is to retry with exponential backoff, waiting for the cold-start to complete. A 503 that persists beyond 90 seconds indicates a genuine deployment issue (model load failure, Docker image corruption, insufficient GPU memory); a 503 that resolves within 60 seconds is a normal cold-start.

# Baseten cold-start timeline for a typical ML model deployment:
#
# T+0s:   Request arrives at scaled-down deployment (min_scale=0)
#         Baseten autoscaler receives trigger event
#         Response: HTTP 503 Service Unavailable
#
# T+0s to T+15s:
#         Baseten provisions new GPU instance
#         Pulls model Docker image to instance
#         Response: HTTP 503 (still in cold-start)
#
# T+15s to T+40s:
#         GPU instance initialized
#         Model weights loaded from storage into GPU VRAM
#         Truss server process starting
#         Response: HTTP 503 (still in cold-start)
#
# T+40s to T+60s:
#         Truss server health check passes
#         Baseten load balancer marks replica as healthy
#         Response: HTTP 200 (cold-start complete)
#
# For Eastfield MLOps Ltd., the cold-start took ~45 seconds on average.
# A client with 3 retries at 2s/4s/8s delays (total wait: 14 seconds)
# would retry at T+2s, T+6s, T+14s — all within the cold-start window.
# A 4th retry at T+20s typically still hits cold-start.
# The fix uses delays of 2s/4s/8s for a total wait of ~22 seconds by retry 3,
# which is still within the cold-start window — but the timeout was also increased
# to 60 seconds, so the 4th attempt at T+22s often waits long enough for the
# cold-start to finish mid-request.
#
# For deployments with longer cold-starts (large models, slow image pulls),
# increase delays to 5s/10s/20s or use min_scale=1 for zero cold-start latency.

The fix: exponential backoff retry on 503

# classifier.py — FIXED
# Eastfield MLOps Ltd.
# requests 2.31.0, Python 3.11

import requests
import time
import os

BASETEN_API_KEY = os.environ["BASETEN_API_KEY"]
BASETEN_MODEL_ID = "abcd1234"
BASETEN_URL = f"https://model-{BASETEN_MODEL_ID}.api.baseten.co/production/predict"

# Retry delays for 503 cold-start: 2s, 4s, 8s.
# Total maximum wait before final attempt: 2+4+8 = 14s elapsed.
# Combined with the 60s request timeout, this covers Baseten's 30-60s cold-start.
BASETEN_503_RETRY_DELAYS = [2, 4, 8]

def classify_document(document_text: str) -> dict:
    last_error = None

    for attempt, delay in enumerate([0] + BASETEN_503_RETRY_DELAYS):
        if delay > 0:
            # Wait before retry — give Baseten time to complete cold-start.
            time.sleep(delay)

        try:
            response = requests.post(
                BASETEN_URL,
                headers={
                    "Authorization": f"Api-Key {BASETEN_API_KEY}",
                    "Content-Type": "application/json",
                },
                json={"text": document_text},
                timeout=60,  # increased from 30s — allow cold-start to complete mid-request
            )

            if response.status_code == 503:
                # ← FIX: 503 is a transient Baseten cold-start signal, not a permanent error.
                # Log and retry with backoff. On the final attempt, fall through to raise.
                if attempt < len(BASETEN_503_RETRY_DELAYS):
                    print(
                        f"[INFO] Baseten 503 (cold-start likely). "
                        f"Attempt {attempt + 1}/{len(BASETEN_503_RETRY_DELAYS) + 1}. "
                        f"Retrying in {BASETEN_503_RETRY_DELAYS[attempt]}s."
                    )
                    last_error = requests.HTTPError(
                        f"503 on attempt {attempt + 1}", response=response
                    )
                    continue

            # For all other status codes (200, 4xx, non-503 5xx): raise immediately.
            response.raise_for_status()
            return response.json()

        except requests.exceptions.Timeout:
            # Timeout during cold-start: treat as transient, retry.
            if attempt < len(BASETEN_503_RETRY_DELAYS):
                print(
                    f"[INFO] Baseten timeout (cold-start likely). "
                    f"Attempt {attempt + 1}. Retrying."
                )
                last_error = requests.exceptions.Timeout("Timeout during cold-start")
                continue
            raise

    # All retries exhausted — cold-start did not complete within retry window.
    raise requests.HTTPError(
        f"Baseten 503 persisted after {len(BASETEN_503_RETRY_DELAYS) + 1} attempts. "
        f"Cold-start may have failed. Check Baseten deployment status.",
        response=getattr(last_error, "response", None),
    ) from last_error


# Alternative: use the tenacity library for cleaner retry logic.
# tenacity is production-grade and handles retry state, logging, and jitter.

from tenacity import (
    retry,
    stop_after_attempt,
    wait_exponential,
    retry_if_exception,
)

def _is_baseten_503(exc: Exception) -> bool:
    return (
        isinstance(exc, requests.HTTPError)
        and exc.response is not None
        and exc.response.status_code == 503
    )

@retry(
    retry=retry_if_exception(_is_baseten_503),
    wait=wait_exponential(multiplier=1, min=2, max=8),
    stop=stop_after_attempt(4),
)
def classify_document_tenacity(document_text: str) -> dict:
    response = requests.post(
        BASETEN_URL,
        headers={"Authorization": f"Api-Key {BASETEN_API_KEY}"},
        json={"text": document_text},
        timeout=60,
    )
    response.raise_for_status()  # raises HTTPError on 503 → tenacity retries
    return response.json()

Why eight weeks of infrastructure investigation were a dead end

The Eastfield team filed four Baseten support tickets over eight weeks describing “intermittent 503 errors”. Baseten’s support responses confirmed that the deployment was healthy and that 503 during cold-start was expected behavior for min_scale=0 deployments. The team did not act on this response because they believed their traffic patterns should not cause cold-starts — they had continuous document uploads throughout the day. What they missed: the overnight idle period. Document uploads from US clients stopped at 9 PM EST and resumed at 8 AM EST. Eleven hours of no traffic triggered the autoscaler’s scale-down. The first morning upload always hit a cold replica.

The retainer developer identified the cold-start pattern in five minutes by correlating the error timestamps against the Baseten deployment scaling event log. Every 503 occurred within 60 seconds of the first morning upload. Every error window ended exactly when the replica became healthy. The pattern was unmistakable: the errors were not random, not infrastructure-wide, and not related to model packaging — they were clock-predictable cold-start events.

The retainer developer also noted an additional factor: the 30-second request timeout was shorter than the cold-start duration. Even if a retry had been attempted, the timeout would have expired before the replica became healthy. The fix required both retry logic and a timeout increase to 60 seconds.

Validation

# Validation benchmark (October 29, 2026)
# Cold-start simulation: wait for scale-down, then send 200 classification requests
# requests 2.31.0, Python 3.11, custom Truss model on Baseten (min_scale=0)

# Before fix (no retry, 30s timeout):
# Documents classified successfully:          146 / 200  (73%)
# Documents failing with HTTP 503:             54 / 200  (27%)  ← unusually high: simulated burst cold-start
# Documents surfacing error to caller:         54 / 200  (27%)  ← all 503s are permanent failures
# Average daily 503 rate in production:       2-3%               ← 30-60s window per morning
# Duration: 8 weeks February–October 2026
# User impact: contract upload returns 'classification failed' with no retry option

# After fix (exponential backoff 2s/4s/8s on 503; 60s timeout):
# Documents classified successfully:          200 / 200 (100%)
# Documents requiring 1 retry (503→200):       38 / 200  (19%)  ← cold-start resolved by retry
# Documents requiring 2 retries (503→503→200): 14 / 200   (7%)  ← cold-start slightly longer
# Documents requiring 3+ retries:              2 / 200   (1%)  ← cold-start at edge of window
# User-facing errors:                           0 / 200   (0%)
# Average cold-start wait (p50):              42s   (time from first 503 to first 200)
# Average cold-start wait (p95):              58s
# Average latency for warm requests (p50):   380ms  (no cold-start)

# Production result (Eastfield MLOps Ltd.):
# Morning 503 error rate:  2-3% → 0%  (all cold-start 503s resolved by retry)
# User-facing errors:      2-3% → 0%
# Truss packaging: no changes (was never the cause)
# Docker image: no changes (was never the cause)
# Network configuration: no changes (was never the cause)
# min_scale=0: retained (cost savings preserved; cold-start handled by client retry)
# 2.0h retainer engagement.
# Triage: 0.4h — correlate 503 timestamps with Baseten scaling events; confirm cold-start pattern.
# Root cause: 0.4h — confirm min_scale=0; identify no retry logic; confirm 30s timeout < cold-start duration.
# Fix: 0.6h — implement exponential backoff on 503; increase timeout to 60s; add cold-start logging.
# Validation: 0.6h — cold-start simulation; 200-request batch; confirm 0% user-facing errors; add CI cold-start test.

Related retainer posts

Track retainer hours without a client portal

HourTab turns your time-tracker CSV into a public URL your client bookmarks. No login. No portal. CSV in, URL out.

See pricing →