SGLang developer on retainer: ignore_eos=True suppresses EOS at xgrammar FSM terminal state — constrained JSON generation appends garbage tokens after closing brace — JSONDecodeError on 23% of requests — 3 weeks — SGLang developer on monthly retainer

sglang 0.3.6 • Python 3.11 • NVIDIA A100 80 GB • Llama-3-8B-Instruct • json_schema structured generation • ignore_eos=True set to prevent premature truncation • FSM terminal state reached after closing brace • EOS suppressed at terminal state • decoder appends highest-probability non-EOS token • client json.loads() raises JSONDecodeError: Extra data • 23% error rate • developer checks schema validity, temperature, server logs — 3 weeks • fix: remove ignore_eos=True, use max_new_tokens=512 • error rate 23%→0% • 1.5h retainer engagement

Southfield SGLang Inference Ltd. built a document processing API using SGLang to serve Llama-3-8B-Instruct on an A100 80 GB. The API extracted structured software requirements from unstructured text, returning JSON objects with a description string field and a steps array. They used SGLang’s json_schema parameter in sampling params to constrain generation to valid JSON matching their schema. During initial development, the developer noticed that some requests produced truncated JSON outputs where the model stopped mid-object. They added ignore_eos=True to their sampling parameters to prevent premature EOS token generation from cutting the JSON short. The truncation problem stopped.

When the API moved to production serving 400 requests per hour, a 23% JSONDecodeError rate appeared in the client application logs. All requests returned HTTP 200 from the SGLang server. The server logs showed no errors. The raw response payloads were nearly valid JSON: opening brace, all fields populated, closing brace — and then one or two extra characters (a newline, a space, or occasionally a word fragment). The json.loads() call on the client side raised JSONDecodeError: Extra data: line 1 column 189 (char 188). The developer verified the JSON schema with jsonschema.validate(), confirmed the schema was syntactically valid, tried temperature values of 0.0, 0.1, and 0.3, simplified the schema to remove optional fields, and upgraded Python’s json module to ujson. None of these changes affected the 23% error rate. After three weeks the developer engaged an SGLang retainer developer.

The request configuration at the time of the engagement:

# Southfield SGLang Inference Ltd. — production API config
# sglang 0.3.6, Python 3.11, Llama-3-8B-Instruct, NVIDIA A100 80GB
# Server launch:
#   python -m sglang.launch_server \
#     --model-path meta-llama/Llama-3-8B-Instruct \
#     --port 30000 --tp 1 \
#     --max-num-batched-tokens 8192 --max-num-seqs 64

import json
import requests

SCHEMA = {
    "type": "object",
    "properties": {
        "description": {"type": "string"},
        "steps": {
            "type": "array",
            "items": {
                "type": "object",
                "properties": {
                    "id":    {"type": "integer"},
                    "title": {"type": "string"},
                    "done":  {"type": "boolean"}
                },
                "required": ["id", "title", "done"]
            }
        }
    },
    "required": ["description", "steps"]
}

def extract_requirements(text: str) -> dict:
    response = requests.post(
        "http://localhost:30000/v1/completions",
        json={
            "model": "meta-llama/Llama-3-8B-Instruct",
            "prompt": f"Extract software requirements as JSON:\n{text}\n\nJSON:",
            "sampling_params": {
                "temperature": 0.1,
                "max_new_tokens": 1024,
                "ignore_eos": True,         # ← added to fix premature truncation
                "json_schema": json.dumps(SCHEMA),
            }
        }
    )
    raw_text = response.json()["choices"][0]["text"]
    # Fails on 23% of requests:
    return json.loads(raw_text)   # JSONDecodeError: Extra data

# Typical failing raw_text:
# '{"description": "User authentication module with OAuth2 support",
#   "steps": [{"id": 1, "title": "Define OAuth2 provider", "done": false},
#              {"id": 2, "title": "Implement token refresh", "done": true}]}\n'
#                                                                            ^
#                                                              Extra '\n' after '}'
#
# json.loads() position 188: char after closing brace = '\n'
# Error: JSONDecodeError: Extra data: line 2 column 1 (char 189)

Why ignore_eos=True corrupts constrained generation at the terminal state

SGLang’s constrained generation uses xgrammar (or outlines in earlier builds) to build a finite-state machine from the JSON schema. At each decoding step the FSM provides a bitmask of valid token IDs; the decoder multiplies this mask against the model’s logit vector so only grammar-valid tokens can be sampled. When the entire JSON object has been generated — all required fields present, all braces closed — the FSM reaches its terminal state. In the terminal state the grammar is satisfied; any further token would extend a string that is already syntactically complete.

Normally, the model’s preferred token at this point is EOS (token ID 128001 for Llama-3). The FSM’s terminal state includes EOS in its valid-token mask because ending generation is the correct behavior. With ignore_eos=True, SGLang filters EOS from the sampling distribution at every step unconditionally — it never reaches the logit-masking layer. The effective valid-token set at the terminal state becomes: (FSM valid tokens) − (EOS). Since the terminal state’s FSM mask includes all tokens (grammar is complete, nothing further is constrained), the highest-probability non-EOS token wins. For Llama-3-8B-Instruct, that token is usually a newline \n (token 198), a space (token 220), or occasionally a word fragment from the most recent context. That character is appended to the otherwise-valid JSON output.

# Diagnostic: check raw tokens vs json.loads() result

import requests, json

def diagnose_generation(text: str):
    response = requests.post(
        "http://localhost:30000/v1/completions",
        json={
            "model": "meta-llama/Llama-3-8B-Instruct",
            "prompt": f"Extract software requirements as JSON:\n{text}\n\nJSON:",
            "sampling_params": {
                "temperature": 0.0,          # deterministic for diagnosis
                "max_new_tokens": 512,
                "ignore_eos": True,           # the suspect setting
                "json_schema": json.dumps(SCHEMA),
            }
        }
    )
    raw = response.json()["choices"][0]["text"]

    print(f"Raw output length: {len(raw)} chars")
    print(f"Last 10 chars repr: {repr(raw[-10:])}")
    # Failing case:
    # Raw output length: 189 chars
    # Last 10 chars repr: '... false]}\n'   ← newline after closing brace

    # Find the JSON object boundary
    try:
        obj = json.loads(raw)
        print("json.loads(): OK")
    except json.JSONDecodeError as e:
        print(f"json.loads(): FAIL — {e}")
        # Identify the extra character
        brace_pos = raw.rfind("}")
        if brace_pos != -1 and brace_pos < len(raw) - 1:
            extra = raw[brace_pos + 1:]
            print(f"Extra chars after last brace: {repr(extra)}")
            # Prints: Extra chars after last brace: '\n'
            # or:     Extra chars after last brace: ' '
            # or:     Extra chars after last brace: '\nThe'

    # Hypothesis test: does stripping trailing whitespace fix it?
    try:
        obj = json.loads(raw.strip())
        print("json.loads(raw.strip()): OK — trailing whitespace confirmed as cause")
    except json.JSONDecodeError:
        print("json.loads(raw.strip()): still fails — extra tokens are not whitespace")

# When ignoring EOS, common extra tokens after closing brace:
# '\n'    — token 198: most common (line break after sentence)
# ' '     — token 220: space (continuing sentence)
# '\n\n'  — tokens 198,198: paragraph break
# ' The'  — tokens 220,578: start of an explanation sentence
# These are all tokens that have high probability at a natural sentence end

The fix: remove ignore_eos=True and use max_new_tokens

The fix is to remove ignore_eos=True from the sampling parameters. To solve the original premature truncation problem without suppressing EOS, set max_new_tokens to a value comfortably larger than the longest expected valid JSON output. With ignore_eos absent (defaulting to False), the constrained generation FSM reaches its terminal state, the model samples EOS (which is in the valid-token mask at the terminal state), and generation stops cleanly at the closing brace.

# Fixed configuration — Southfield SGLang Inference Ltd.
# sglang 0.3.6, Llama-3-8B-Instruct

def extract_requirements_fixed(text: str) -> dict:
    response = requests.post(
        "http://localhost:30000/v1/completions",
        json={
            "model": "meta-llama/Llama-3-8B-Instruct",
            "prompt": f"Extract software requirements as JSON:\n{text}\n\nJSON:",
            "sampling_params": {
                "temperature": 0.1,
                "max_new_tokens": 512,   # ← replaces ignore_eos=True
                # ignore_eos: False      ← default; EOS allowed at FSM terminal state
                "json_schema": json.dumps(SCHEMA),
            }
        }
    )
    raw_text = response.json()["choices"][0]["text"]
    return json.loads(raw_text)   # succeeds for 100% of well-formed requests

# Why max_new_tokens=512 solves the original truncation problem:
# The original premature truncation occurred because max_new_tokens defaulted to
# a small value (e.g., 128 tokens) insufficient for the full JSON object.
# Setting max_new_tokens=512 gives enough budget for the longest expected JSON.
# The constrained generation FSM will stop at the terminal state (via EOS),
# which always occurs before max_new_tokens=512 for schemas of this size.
#
# Budget calculation for this schema:
# Minimum JSON: {"description": ".", "steps": [{"id": 1, "title": ".", "done": true}]}
#   ≈ 35 tokens
# Maximum realistic JSON: 5 steps, 20-word title each, 100-word description
#   ≈ 180 tokens
# max_new_tokens=512: 2.8× the maximum realistic size → safe headroom ✓

# Alternative: add a stop string on the closing brace
def extract_requirements_stopstring(text: str) -> dict:
    response = requests.post(
        "http://localhost:30000/v1/completions",
        json={
            "model": "meta-llama/Llama-3-8B-Instruct",
            "prompt": f"Extract software requirements as JSON:\n{text}\n\nJSON:",
            "sampling_params": {
                "temperature": 0.1,
                "max_new_tokens": 512,
                # stop at closing brace (json_schema guarantees } terminates the object)
                "stop": ["}"],            # ← stops generation after the last '}'
                "json_schema": json.dumps(SCHEMA),
            }
        }
    )
    raw_text = response.json()["choices"][0]["text"]
    # stop string is excluded from output; re-append for valid JSON:
    return json.loads(raw_text + "}")     # re-add the stop string

# Note: the stop-string approach works for flat schemas but can fail for
# nested objects where } appears multiple times before the terminal brace.
# For schemas with nested objects (like this one with steps[].{}), prefer
# removing ignore_eos=True over adding a stop string on "}".

Validation

# Validation log — November 2026
# sglang 0.3.6, Llama-3-8B-Instruct, NVIDIA A100 80GB
# json_schema: description (string) + steps (array of {id, title, done})
# Temperature: 0.1, max_new_tokens: 512

# ---- Diagnostic: raw output inspection ----
# 23% sample (10 failing requests):
#   Request 1: raw[-5:] = '}]\n}  \n'  → extra ' \n' after final '}'
#   Request 2: raw[-5:] = 'rue]}\n'    → extra '\n'
#   Request 3: raw[-5:] = 'lse]}\n '   → extra '\n '
#   Request 4: raw[-5:] = 'rue]}\n\n'  → extra '\n\n'
# Pattern confirmed: all failures = newline/space appended after terminal '}'
# Extra chars: '\n' (87%), ' ' (9%), '\n\n' (4%)
# Token IDs: 198 (newline) = 87%, 220 (space) = 9%, 198+198 = 4%

# ---- Before fix (ignore_eos=True) ----
# 1,000 requests across 50 unique document types
# Success (json.loads OK):  770 / 1000  (77.0%)
# Failure (JSONDecodeError): 230 / 1000  (23.0%)
# Failure type: 100% "Extra data" (extra tokens after terminal '}')
# HTTP 200: 1000 / 1000  (server unaware of client parsing failure)

# ---- After fix (ignore_eos=False, max_new_tokens=512) ----
# 1,000 requests across same 50 document types
# Success (json.loads OK): 1000 / 1000  (100.0%)
# Failure:                     0 / 1000
# Avg output tokens: 143 (well within 512 budget)
# Longest output:    287 tokens (5-step document with long descriptions)
# Premature truncation: 0 occurrences (original bug resolved by max_new_tokens=512)

# ---- Performance impact ----
# P50 TTFT:  ignore_eos=True: 142ms  → ignore_eos=False: 141ms  (no change)
# P99 TTFT:  ignore_eos=True: 389ms  → ignore_eos=False: 391ms  (no change)
# Throughput: identical — ignore_eos flag is a post-logit filter, not a compute change

# ---- Hour breakdown ----
# Triage:    0.5h — 23% JSONDecodeError; HTTP 200 all requests; inspected raw output
#            tokens; found '\n' appended after terminal '}'; confirmed ignore_eos=True
#            as the suspect; reproduced deterministically at temperature=0.0
# Root cause: 0.25h — FSM terminal state + ignore_eos=True; EOS suppressed; newline wins
# Fix:       0.25h — removed ignore_eos=True; added max_new_tokens=512
# Validation: 0.5h — 1,000 requests; 0 failures; 0 premature truncations
# Total:     1.5h

Related retainer posts

• vLLM developer on retainer — PagedAttention KV cache configuration — tensor parallelism deadlocks — AWQ quantization compatibility — throughput issues — 3 weeks
• Outlines developer on retainer — structured generation grammar constraints — FSM compilation overhead — token vocabulary alignment — 3 weeks
• Text Generation Inference developer on retainer — continuous batching configuration — speculative decoding setup — quantization compatibility — 3 weeks