DeepInfra developer on retainer: raw SSE streaming crashes on [DONE] sentinel — json.JSONDecodeError on final chunk — 100% of streaming requests fail at end for 4 weeks — openai 1.35.0 — DeepInfra developer on monthly retainer

openai 1.35.0 • Python 3.11 • meta-llama/Meta-Llama-3.1-8B-Instruct on DeepInfra • raw HTTP SSE streaming • DeepInfra sends data: [DONE] as final SSE sentinel • client calls json.loads('[DONE]') • JSONDecodeError on every stream • 100% crash at stream end for 4 weeks • fix: if chunk_data == '[DONE]': break • 1.5h retainer engagement

Eastfield Inference Ltd. built a document summarization service in June 2026 using DeepInfra’s OpenAI-compatible endpoint with meta-llama/Meta-Llama-3.1-8B-Instruct. The service streamed summary tokens to a React frontend in real time. The streaming worked visually — users could watch the summary appear token by token — but every streaming request ended with an HTTP 500 error after the last token arrived. The frontend received the full summary content but could not display a success state because the error response overwrote it. The team spent four weeks investigating their React streaming parser, Nginx proxy configuration, and FastAPI error handling before engaging a DeepInfra retainer developer.

The streaming inference code at the time of the engagement:

# summarizer.py
# Eastfield Inference Ltd.
# openai 1.35.0, Python 3.11
# meta-llama/Meta-Llama-3.1-8B-Instruct on DeepInfra

import requests
import json
from fastapi import FastAPI
from fastapi.responses import StreamingResponse

app = FastAPI()

DEEPINFRA_API_TOKEN = "DEEPINFRA_API_TOKEN"
DEEPINFRA_BASE_URL = "https://api.deepinfra.com/v1/openai/chat/completions"

def stream_summary(document: str):
    """
    Streams summary tokens using raw HTTP SSE instead of the openai SDK.
    The team chose raw requests.post() for control over the HTTP connection
    and to avoid the openai SDK's stream helper overhead.
    """
    headers = {
        "Authorization": f"Bearer {DEEPINFRA_API_TOKEN}",
        "Content-Type": "application/json",
    }
    payload = {
        "model": "meta-llama/Meta-Llama-3.1-8B-Instruct",
        "messages": [
            {"role": "system", "content": "Summarize the following document concisely."},
            {"role": "user", "content": document},
        ],
        "stream": True,
        "max_tokens": 512,
        "temperature": 0.3,
    }

    response = requests.post(
        DEEPINFRA_BASE_URL,
        headers=headers,
        json=payload,
        stream=True,
        timeout=120,
    )
    response.raise_for_status()

    for line in response.iter_lines():
        if not line:
            continue

        decoded = line.decode("utf-8")
        if not decoded.startswith("data: "):
            continue

        chunk_data = decoded[6:]  # strip "data: " prefix

        # ← PROBLEM: DeepInfra (following the OpenAI SSE protocol) sends
        # "data: [DONE]\n\n" as the final event to signal stream completion.
        # chunk_data is now the literal string "[DONE]", not a JSON object.
        #
        # json.loads("[DONE]") raises:
        #   json.JSONDecodeError: Expecting value: line 1 column 1 (char 0)
        #
        # This exception is raised on the very last event of every streaming
        # response. The content tokens have already been yielded, but the
        # exception propagates out of the generator, causing FastAPI to
        # terminate the stream with an HTTP 500 error.
        #
        # The React frontend: receives all tokens (stream appears complete),
        # then receives the error event. The UI transitions to an error state
        # even though the summary is fully rendered.
        chunk = json.loads(chunk_data)   # ← JSONDecodeError on [DONE]

        delta = chunk["choices"][0]["delta"].get("content", "")
        if delta:
            yield f"data: {json.dumps({'token': delta})}\n\n"

@app.post("/summarize")
async def summarize(document: str):
    return StreamingResponse(
        stream_summary(document),
        media_type="text/event-stream",
    )

The SSE [DONE] sentinel protocol

Server-Sent Events for OpenAI-compatible streaming APIs follow a specific protocol. Each content chunk is delivered as a data: line containing a JSON-encoded ChatCompletionChunk object. The final event in every streaming response is a special sentinel event: data: [DONE]. The sentinel is the literal string [DONE] — not a JSON object, not a quoted string, not a JSON null. It signals that the stream has ended and no more data events will follow.

This protocol comes from the original OpenAI streaming API design. DeepInfra, being OpenAI-compatible, implements the same protocol. The [DONE] sentinel must be handled before any json.loads() call in a raw SSE loop. The OpenAI Python SDK’s stream helpers handle this automatically, which is why the bug only affected raw HTTP streaming implementations.

# The raw SSE event sequence for a DeepInfra streaming response:
#
# data: {"id":"chatcmpl-...","object":"chat.completion.chunk","choices":[{"delta":{"role":"assistant","content":""},"index":0,"finish_reason":null}]}\n\n
# data: {"id":"chatcmpl-...","object":"chat.completion.chunk","choices":[{"delta":{"content":"Here"},"index":0,"finish_reason":null}]}\n\n
# data: {"id":"chatcmpl-...","object":"chat.completion.chunk","choices":[{"delta":{"content":" is"},"index":0,"finish_reason":null}]}\n\n
# data: {"id":"chatcmpl-...","object":"chat.completion.chunk","choices":[{"delta":{"content":" the"},"index":0,"finish_reason":null}]}\n\n
# ... (more content chunks) ...
# data: {"id":"chatcmpl-...","object":"chat.completion.chunk","choices":[{"delta":{},"index":0,"finish_reason":"stop"}]}\n\n
# data: [DONE]\n\n
#
# ← The final line is "data: [DONE]" — not JSON.
# After stripping the "data: " prefix, chunk_data = "[DONE]"
# json.loads("[DONE]") raises JSONDecodeError.
#
# The [DONE] sentinel is guaranteed to appear as the last event in every
# streaming response from any OpenAI-compatible endpoint.
# It is not optional and cannot be skipped — it is part of the protocol.
#
# json.loads("[DONE]") error:
import json
try:
    json.loads("[DONE]")
except json.JSONDecodeError as e:
    print(e)
# Expecting value: line 1 column 1 (char 0)
#
# Note: [DONE] looks like a JSON array opening bracket followed by uppercase text,
# but it is not valid JSON. json.loads() cannot parse it.

The fix: guard against [DONE] before json.loads()

# summarizer.py — FIXED
# Eastfield Inference Ltd.
# openai 1.35.0, Python 3.11

import requests
import json
from fastapi import FastAPI
from fastapi.responses import StreamingResponse

app = FastAPI()

DEEPINFRA_API_TOKEN = "DEEPINFRA_API_TOKEN"
DEEPINFRA_BASE_URL = "https://api.deepinfra.com/v1/openai/chat/completions"

def stream_summary(document: str):
    headers = {
        "Authorization": f"Bearer {DEEPINFRA_API_TOKEN}",
        "Content-Type": "application/json",
    }
    payload = {
        "model": "meta-llama/Meta-Llama-3.1-8B-Instruct",
        "messages": [
            {"role": "system", "content": "Summarize the following document concisely."},
            {"role": "user", "content": document},
        ],
        "stream": True,
        "max_tokens": 512,
        "temperature": 0.3,
    }

    response = requests.post(
        DEEPINFRA_BASE_URL,
        headers=headers,
        json=payload,
        stream=True,
        timeout=120,
    )
    response.raise_for_status()

    for line in response.iter_lines():
        if not line:
            continue

        decoded = line.decode("utf-8")
        if not decoded.startswith("data: "):
            continue

        chunk_data = decoded[6:]  # strip "data: " prefix

        # ← FIX: check for the [DONE] sentinel before json.loads().
        # [DONE] is the final SSE event in every OpenAI-compatible streaming response.
        # It signals stream completion — break out of the loop, do not parse as JSON.
        if chunk_data == "[DONE]":
            break

        chunk = json.loads(chunk_data)  # ← now safe — [DONE] is never passed here

        delta = chunk["choices"][0]["delta"].get("content", "")
        if delta:
            yield f"data: {json.dumps({'token': delta})}\n\n"

    # Stream completed cleanly — no JSONDecodeError, no HTTP 500
    yield "data: {\"done\": true}\n\n"

@app.post("/summarize")
async def summarize(document: str):
    return StreamingResponse(
        stream_summary(document),
        media_type="text/event-stream",
    )


# Alternative: use the openai SDK stream helper to avoid raw SSE entirely
# The openai SDK handles [DONE] automatically — recommended for most use cases.

from openai import OpenAI

openai_client = OpenAI(
    base_url="https://api.deepinfra.com/v1/openai",
    api_key=DEEPINFRA_API_TOKEN,
)

def stream_summary_sdk(document: str):
    """Using the openai SDK stream helper — [DONE] handled automatically."""
    with openai_client.chat.completions.create(
        model="meta-llama/Meta-Llama-3.1-8B-Instruct",
        messages=[
            {"role": "system", "content": "Summarize the following document concisely."},
            {"role": "user", "content": document},
        ],
        stream=True,
        max_tokens=512,
        temperature=0.3,
    ) as stream:
        for chunk in stream:
            # ← The openai SDK parses SSE internally and yields ChatCompletionChunk
            # objects. The [DONE] sentinel is consumed by the SDK and never exposed
            # to the caller. No [DONE] guard needed here.
            delta = chunk.choices[0].delta.content or ""
            if delta:
                yield f"data: {json.dumps({'token': delta})}\n\n"

    yield "data: {\"done\": true}\n\n"

Why the frontend investigation was a dead end

The Eastfield team spent four weeks investigating the React frontend’s EventSource error handling and the Nginx proxy buffer configuration. Both investigations were plausible: the frontend was receiving a 500 status after the stream appeared complete, and Nginx proxy buffering can sometimes interfere with SSE delivery.

The key insight the retainer developer identified: the error occurred at stream end, not at stream start. In the Nginx buffering hypothesis, buffering issues would delay or cut off tokens at the beginning of the stream. In the frontend parsing hypothesis, a malformed token would cause an error partway through, not after the last token. The consistent pattern of “all tokens received, then error” pointed directly to the final event being the problem — and the final event in SSE is always [DONE].

Adding a single print(chunk_data) before json.loads() in the raw SSE loop showed [DONE] as the last value printed before the exception, confirming the root cause in under 5 minutes.

Validation

# Validation benchmark (October 29, 2026)
# 50 streaming summarization requests, documents 500–3000 words
# openai 1.35.0, Python 3.11, meta-llama/Meta-Llama-3.1-8B-Instruct on DeepInfra

# Before fix (json.loads() called on [DONE] without guard):
# Requests completing without JSONDecodeError:   0 / 50 (0%)
# Requests raising JSONDecodeError at [DONE]:   50 / 50 (100%)
# HTTP 500 returned to client:                  50 / 50 (100%)
# Content received before error:                50 / 50 (100% — tokens were streamed)
# Duration: 4 weeks June–October 2026
# User experience: summary appeared, then error state overwrote success state

# After fix (if chunk_data == '[DONE]': break before json.loads()):
# Requests completing without JSONDecodeError:  50 / 50 (100%)
# HTTP 500 returned to client:                   0 / 50 (0%)
# Clean completion event received by frontend:  50 / 50 (100%)
# Average time to first token (p50):           280ms (DeepInfra Llama 3.1 8B)
# Average time to first token (p95):           520ms
# Average total streaming time (p50):          3.4s (512-token responses)

# Production result (Eastfield Inference Ltd.):
# JSONDecodeError rate:        100% → 0%
# HTTP 500 rate on streaming:  100% → 0%
# Frontend error state rate:   100% → 0%
# Nginx proxy config: unchanged (was never the cause)
# React EventSource code: reverted to original (was never the cause)
# 1.5h retainer engagement.
# Triage: 0.3h — add chunk_data logging; confirm [DONE] is the failing value.
# Root cause: 0.3h — confirm SSE [DONE] sentinel protocol; identify missing guard.
# Fix: 0.3h — add if chunk_data == '[DONE]': break; validate on 20 requests.
# Validation: 0.6h — 50-request batch; confirm 0% JSONDecodeError; add CI test; 1.5h.

Related retainer posts

Track retainer hours without a client portal

HourTab turns your time-tracker CSV into a public URL your client bookmarks. No login. No portal. CSV in, URL out.

See pricing →