Jina AI developer on retainer: missing Authorization header triggers free-tier 20 RPM limit — 80% HTTP 429 on document processing pipeline for 6 weeks — requests 2.31.0 — Jina AI developer on monthly retainer

requests 2.31.0 • Python 3.11 • Jina AI Reader API at r.jina.ai • missing Authorization: Bearer header • client routes to anonymous free-tier 20 RPM • pipeline fires 100+ RPM • 80% HTTP 429 for 6 weeks • tight exponential backoff exhausts retries within 60-second window • fix: Authorization: Bearer {JINA_API_KEY} (500 RPM tier) + Retry-After header compliance • 1.5h retainer engagement

Eastfield Document AI Ltd. built a document processing pipeline in April 2026 using the Jina AI Reader API to extract clean text from 1,200 PDF and web page URLs for a legal research tool. The pipeline sent GET requests to https://r.jina.ai/{target_url} using the requests library with 8 concurrent workers, targeting the 1,200 documents over a 10-minute batch window. From the first day of production, 80% of requests received HTTP 429 Too Many Requests. The team added exponential backoff (1s, 2s, 4s delays between retries), which consumed all three retry attempts within the same 60-second rate limit window, resulting in still-failed requests. The pipeline achieved only 20% document extraction success for six weeks while the team investigated network infrastructure, IP allowlisting, and Jina AI account configuration.

The Jina AI Reader API applies rate limits based on authentication state: requests without an Authorization: Bearer {JINA_API_KEY} header are treated as anonymous free-tier requests at 20 requests per minute, regardless of whether the caller has a paid Jina AI account. The pipeline was firing 100+ requests per minute from 8 concurrent workers, but every request lacked the Authorization header. The first 20 requests each minute succeeded; the remaining 80+ received HTTP 429. The team had a paid Jina AI API key, had logged into the dashboard, and believed their account was configured correctly — but never transferred the key into the client’s request headers.

The document extraction client code at the time of the engagement:

# document_extractor.py
# Eastfield Document AI Ltd.
# requests 2.31.0, Python 3.11
# Jina AI Reader API at https://r.jina.ai/

import requests
import time
import os

JINA_API_KEY = os.environ["JINA_API_KEY"]  # ← Key exists in env; never used below

JINA_BASE_URL = "https://r.jina.ai/"

# ← PROBLEM: No Authorization header. Without Authorization: Bearer {JINA_API_KEY},
# Jina AI Reader treats ALL requests as anonymous free-tier: 20 RPM.
# The paid JINA_API_KEY is in the environment but never sent in request headers.
# 8 concurrent workers fire 100+ RPM against the 20 RPM anonymous limit.
# 80+ requests per 60-second window receive HTTP 429.
HEADERS = {
    "X-Return-Format": "markdown",
    # ← Missing: "Authorization": f"Bearer {JINA_API_KEY}"
}


def extract_document(url: str, max_retries: int = 3) -> str:
    """Extract clean markdown text from URL via Jina AI Reader."""
    for attempt in range(max_retries):
        response = requests.get(
            f"{JINA_BASE_URL}{url}",
            headers=HEADERS,
            timeout=30,
        )

        if response.status_code == 200:
            return response.text

        if response.status_code == 429:
            # ← WRONG backoff: 1s, 2s, 4s delays fire all retries within
            # the same 60-second rate limit window. All retries also get 429.
            # The Retry-After response header specifies when to retry (30-60s).
            # Ignoring it and retrying in 1-4s wastes all retry budget.
            wait = 2 ** attempt          # 1s, 2s, 4s
            time.sleep(wait)
            continue

        response.raise_for_status()

    raise RuntimeError(f"Failed to extract {url} after {max_retries} attempts")

Why six weeks passed without a diagnosis

The team’s first hypothesis was IP-based rate limiting. Jina AI’s documentation mentioned that rate limits applied per IP address on the free tier. The team moved the pipeline to a different cloud region and provisioned a new IP address. HTTP 429 continued at the same rate. They tried NAT gateway rotation to cycle through multiple IP addresses, which briefly reduced the rate of 429 responses but did not eliminate them.

The second hypothesis was account tier misconfiguration. The team verified their Jina AI account showed a paid plan in the dashboard. They contacted Jina AI support, who confirmed the account was in good standing and had the authenticated tier active. The support ticket asked them to “confirm you are sending the API key in request headers.” The team interpreted this as a boilerplate question and responded that their API key was “configured in the environment.” Neither side caught that the key was never actually sent with requests.

The third hypothesis was that HTTP 429 indicated a Jina AI service degradation affecting their account. The team reduced concurrency from 8 to 2 workers and reduced the batch size from 1,200 to 100 documents. With 2 workers and 100 documents, they stayed under 20 RPM and the 429 errors disappeared. They interpreted this as confirmation of a Jina AI service capacity issue and filed another support ticket asking for a rate limit increase. The rate limit increase request was denied because the account was on the anonymous tier — not the authenticated tier.

The retainer developer examined the request headers being sent and immediately identified the absent Authorization header. Checking the Jina AI dashboard’s “API Keys” section confirmed an active key existed. The X-RateLimit-Limit header on a 429 response showed 20 — the anonymous tier limit. Adding the Authorization: Bearer header to every request moved the caller to the 500 RPM authenticated tier. The retainer developer also fixed the backoff to read the Retry-After header value rather than using a fixed 1/2/4-second schedule. Total engagement: 1.5 hours.

The fix: add Authorization: Bearer header and respect Retry-After

# document_extractor.py — FIXED
# Eastfield Document AI Ltd.
# requests 2.31.0, Python 3.11

import requests
import time
import os

JINA_API_KEY = os.environ["JINA_API_KEY"]

JINA_BASE_URL = "https://r.jina.ai/"

# ← FIX: Include Authorization: Bearer header on every request.
# Without it, Jina AI routes requests to anonymous free-tier (20 RPM),
# regardless of paid account status. With it: authenticated tier (500 RPM).
HEADERS = {
    "Authorization": f"Bearer {JINA_API_KEY}",  # ← FIX: authenticated 500 RPM tier
    "X-Return-Format": "markdown",
    "Accept": "application/json",
}


def extract_document(url: str, max_retries: int = 4) -> str:
    """Extract clean markdown text from URL via Jina AI Reader."""
    for attempt in range(max_retries):
        response = requests.get(
            f"{JINA_BASE_URL}{url}",
            headers=HEADERS,
            timeout=30,
        )

        if response.status_code == 200:
            # ← Confirm authenticated tier: X-RateLimit-Limit should be 500.
            rate_limit = response.headers.get("X-RateLimit-Limit", "unknown")
            if rate_limit == "20":
                raise RuntimeError(
                    "Jina AI responding at 20 RPM anonymous tier — "
                    "check that Authorization: Bearer header is being sent"
                )
            return response.text

        if response.status_code == 429:
            # ← FIX: Read Retry-After header and sleep for the specified duration.
            # Retry-After tells us when the rate limit window resets (30–60s typically).
            # The old 1s/2s/4s backoff fired all retries within the window — all 429.
            retry_after = int(response.headers.get("Retry-After", "60"))
            if attempt < max_retries - 1:
                time.sleep(retry_after)   # ← FIX: wait for rate limit reset, not 1s
            continue

        if response.status_code == 402:
            raise RuntimeError(f"Jina AI quota exhausted (HTTP 402) for URL: {url}")

        response.raise_for_status()

    raise RuntimeError(
        f"Failed to extract {url} after {max_retries} attempts (persistent 429)"
    )


def _integration_test():
    """Pre-flight: confirm authenticated 500 RPM tier before batch processing."""
    response = requests.get(
        f"{JINA_BASE_URL}https://example.com",
        headers=HEADERS,
        timeout=30,
    )
    assert response.status_code == 200, f"Jina AI pre-flight failed: HTTP {response.status_code}"
    rate_limit = response.headers.get("X-RateLimit-Limit", "unknown")
    assert rate_limit == "500", (
        f"Jina AI rate limit is {rate_limit} RPM — expected 500 (authenticated). "
        f"Check Authorization: Bearer header is configured correctly."
    )
    print(f"Integration test passed — Jina AI authenticated tier: {rate_limit} RPM")

Validation

# Validation benchmark (October 30, 2026)
# 200 document URL extractions via Jina AI Reader
# requests 2.31.0, Python 3.11, 8 concurrent workers

# Before fix (no Authorization header — anonymous free-tier 20 RPM):
# Requests attempted:               200 / 200
# HTTP 200 (success):                40 / 200  (20%)   ← ~20 RPM * ~2 min batch
# HTTP 429 (rate limited):          160 / 200  (80%)   ← beyond 20 RPM free tier
# Retries that succeeded:             0 / 160  (0%)    ← 1s/2s/4s within 60s window
# Extraction success rate:          20%
# Duration: 6 weeks April–October 2026
# X-RateLimit-Limit header:        "20"        ← anonymous tier confirmed

# After fix (Authorization: Bearer + Retry-After backoff — 500 RPM authenticated):
# Requests attempted:               200 / 200
# HTTP 200 (success):               200 / 200  (100%)
# HTTP 429 (rate limited):            0 / 200  (0%)    ← under 500 RPM limit
# Retries needed:                     0         ← no rate limiting at 8 workers
# Average extraction latency (p50):  ~3.1s
# Average extraction latency (p95):  ~7.4s
# Average extracted text length:    ~8,400 chars per document
# X-RateLimit-Limit header:        "500"        ← authenticated tier confirmed

# Production result (Eastfield Document AI Ltd.):
# Extraction success rate: 20% → 100%
# Pipeline throughput: 100 docs/10min (2 workers) → 1,200 docs/10min (8 workers)
# Fix: Authorization: Bearer {JINA_API_KEY} header + Retry-After backoff
# 1.5h retainer engagement.
# Triage: 0.3h — check request headers; find missing Authorization; read X-RateLimit-Limit (20).
# Root cause: 0.3h — confirm anonymous tier 20 RPM; trace retry backoff window failure.
# Fix: 0.4h — add Authorization header; replace exponential backoff with Retry-After read.
# Validation: 0.5h — 200 documents at 8 workers; 0% HTTP 429; CI integration test added.

Related retainer posts

Track retainer hours without a client portal

HourTab turns your time-tracker CSV into a public URL your client bookmarks. No login. No portal. CSV in, URL out.

See pricing →