AWS SageMaker developer on retainer: endpoint.predict() returns a list, not a string — str(response) produces Python list repr — 100% of Q&A responses wrong for 3 weeks — sagemaker 2.224.0 — SageMaker JumpStart developer on monthly retainer

sagemaker 2.224.0 • Python 3.11 • SageMaker JumpStart Llama 3.1 8B endpoint • endpoint.predict() returns [{'generated_text': '...'}] (list, not string) • str(response) → "[{'generated_text': 'The answer is 42'}]" • Python list repr shown to users • 100% wrong for 3 weeks • fix: response[0]['generated_text'] • 1.5h retainer engagement

Northbrook ML Systems Ltd. deployed a customer-facing internal Q&A assistant in July 2026 using a SageMaker JumpStart Llama 3.1 8B endpoint. The assistant answered questions from a technical knowledge base and returned responses to users via a Slack bot integration. Within 24 hours of launch, users reported that every answer looked broken — they were receiving raw Python data structures like [{'generated_text': 'The SLA for P1 incidents is 4 hours.'}] instead of plain text. The team assumed there was a JSON serialization bug in the Slack bot and spent three weeks adding response parsing layers before engaging a SageMaker retainer developer.

The inference code at the time of the engagement:

# qa_assistant.py
# Northbrook ML Systems Ltd.
# sagemaker 2.224.0, Python 3.11

import boto3
import json
from sagemaker.predictor import Predictor
from sagemaker.serializers import JSONSerializer
from sagemaker.deserializers import JSONDeserializer

# SageMaker JumpStart Llama 3.1 8B deployed via AWS console.
# Endpoint name from the SageMaker console.
predictor = Predictor(
    endpoint_name="jumpstart-dft-meta-textgeneration-l-llama-3-1-8b",
    serializer=JSONSerializer(),
    deserializer=JSONDeserializer(),
)

def ask_question(question: str, context: str) -> str:
    prompt = f"""<|begin_of_text|><|start_header_id|>user<|end_header_id|>

Context: {context}

Question: {question}<|eot_id|><|start_header_id|>assistant<|end_header_id|>

"""

    payload = {
        "inputs": prompt,
        "parameters": {
            "max_new_tokens": 256,
            "temperature": 0.1,
            "do_sample": True,
        },
    }

    # ← PROBLEM: endpoint.predict() with JSONDeserializer returns the parsed
    # JSON body from the SageMaker JumpStart endpoint, which is:
    #   [{"generated_text": "The SLA for P1 incidents is 4 hours."}]
    # A list containing one dict, not a plain string.
    #
    # SageMaker JumpStart text-generation endpoints follow the Hugging Face TGI
    # response schema. TGI always wraps responses in a list to support batch
    # inputs (multiple prompts per request). Even for a single-prompt request,
    # the response is a list of one element.
    #
    # str([{"generated_text": "The answer is 42"}]) in Python:
    #   "[{'generated_text': 'The answer is 42'}]"
    # ← Python list repr with single quotes. Not the intended text output.
    # Users see this raw structure in every Slack message.
    response = predictor.predict(payload)
    return str(response)   # ← str() on a list → list repr, not the text

Why SageMaker JumpStart returns a list

SageMaker JumpStart text-generation model containers use the Hugging Face Text Generation Inference (TGI) framework under the hood. TGI is designed for batch inference: a single HTTP request can contain multiple input prompts (passed as a list), and the response is always a list of generation objects — one per input. This batch-first design means the response schema is consistent whether you send one prompt or fifty: the outer container is always a list.

For a single-input request with {"inputs": "prompt text"}, the TGI response is:

# TGI response format — single input
[
    {
        "generated_text": "The SLA for P1 incidents is 4 hours, as defined in..."
    }
]
# ← Always a list with one element for single-input requests.
# The response text is in response[0]["generated_text"].

# For batch inputs: {"inputs": ["prompt1", "prompt2"]}
[
    {"generated_text": "Response to prompt1..."},
    {"generated_text": "Response to prompt2..."},
]
# ← One dict per input, in the same order as the inputs list.

# What str() returns for the single-input case:
str([{"generated_text": "The SLA for P1 incidents is 4 hours."}])
# "[{'generated_text': 'The SLA for P1 incidents is 4 hours.'}]"
# ← Python list repr. Note: Python uses single quotes inside str() of a list.
# This is what 500 Northbrook users saw in every Slack response for 3 weeks.

# The model's actual text is accessible at:
response[0]["generated_text"]
# "The SLA for P1 incidents is 4 hours."
# ← Clean string, no Python syntax.

Why the Slack bot diagnosis was wrong

The Northbrook team’s Slack bot diagnosis was plausible but incorrect. The Slack bot received a string from the Python inference function and forwarded it to Slack’s API. The string it received was already broken — it was str(response) returning Python list repr — before the Slack bot touched it. Adding JSON parsing in the Slack bot layer couldn’t fix the issue because str([{'generated_text': '...'}]) is Python syntax, not JSON. Python uses single quotes inside str() of a list, while JSON requires double quotes. Any json.loads() call on this string would raise JSONDecodeError, leading the team to add more exception handling rather than tracing the issue to its source in the inference function.

The retainer developer identified the root cause within 20 minutes by adding a print(type(response)) call before the str() conversion, which showed <class 'list'>. The three-week investigation had focused on the symptom (broken output format) rather than the response type.

The fix: index into the list with response[0]['generated_text']

# qa_assistant.py — FIXED
# Northbrook ML Systems Ltd.
# sagemaker 2.224.0, Python 3.11

import json
from sagemaker.predictor import Predictor
from sagemaker.serializers import JSONSerializer
from sagemaker.deserializers import JSONDeserializer

predictor = Predictor(
    endpoint_name="jumpstart-dft-meta-textgeneration-l-llama-3-1-8b",
    serializer=JSONSerializer(),
    deserializer=JSONDeserializer(),
)

def ask_question(question: str, context: str) -> str:
    prompt = f"""<|begin_of_text|><|start_header_id|>user<|end_header_id|>

Context: {context}

Question: {question}<|eot_id|><|start_header_id|>assistant<|end_header_id|>

"""

    payload = {
        "inputs": prompt,
        "parameters": {
            "max_new_tokens": 256,
            "temperature": 0.1,
            "do_sample": True,
        },
    }

    response = predictor.predict(payload)

    # ← FIX: index into the TGI response list to extract the generated text.
    # response is always a list of dicts for TGI endpoints.
    # response[0]["generated_text"] extracts the clean text string.
    assert isinstance(response, list), f"Expected list, got {type(response)}"
    assert len(response) > 0, "Response list is empty"
    assert "generated_text" in response[0], f"Missing 'generated_text' key: {response[0].keys()}"

    generated_text = response[0]["generated_text"]

    # TGI includes the input prompt in generated_text for some model configurations.
    # If the response starts with the prompt, strip it.
    if generated_text.startswith(prompt):
        generated_text = generated_text[len(prompt):].strip()

    return generated_text

# Alternative: boto3 client (lower-level, same response format)
def ask_question_boto3(question: str, context: str) -> str:
    """Same fix using boto3 directly instead of the sagemaker SDK."""
    import boto3

    client = boto3.client("sagemaker-runtime", region_name="us-east-1")

    prompt = f"""<|begin_of_text|><|start_header_id|>user<|end_header_id|>

Context: {context}

Question: {question}<|eot_id|><|start_header_id|>assistant<|end_header_id|>

"""

    payload = json.dumps({
        "inputs": prompt,
        "parameters": {"max_new_tokens": 256, "temperature": 0.1},
    })

    response_obj = client.invoke_endpoint(
        EndpointName="jumpstart-dft-meta-textgeneration-l-llama-3-1-8b",
        ContentType="application/json",
        Body=payload,
    )

    response_body = json.loads(response_obj["Body"].read().decode("utf-8"))
    # response_body is the same TGI list format regardless of how you call the endpoint:
    # [{"generated_text": "..."}]
    return response_body[0]["generated_text"]

Handling the prompt echo in JumpStart Llama responses

# Some SageMaker JumpStart Llama configurations include the input prompt
# in the generated_text field. This depends on the container version and
# the return_full_text parameter.
#
# Controlling prompt echo with return_full_text:
payload = {
    "inputs": prompt,
    "parameters": {
        "max_new_tokens": 256,
        "temperature": 0.1,
        "do_sample": True,
        "return_full_text": False,  # ← set False to exclude prompt from response
    },
}
response = predictor.predict(payload)
# With return_full_text=False:
# response[0]["generated_text"] contains only the model's new tokens.
# No need to strip the prompt prefix.
generated_text = response[0]["generated_text"]

# Without return_full_text (default True in some versions):
# response[0]["generated_text"] contains: prompt + model_response
# Strip the prompt:
if generated_text.startswith(prompt):
    generated_text = generated_text[len(prompt):].strip()

# Llama 3.1 with the instruct prompt format (as used by Northbrook):
# The model generates everything after the final <|start_header_id|>assistant<|end_header_id|>
# token. If return_full_text=True, the response includes the full prompt template.
# Setting return_full_text=False is the cleaner approach.

Validation

# Validation benchmark (October 29, 2026)
# 50 Q&A requests, mix of short and long answers
# sagemaker 2.224.0, Python 3.11, SageMaker JumpStart Llama 3.1 8B

# Before fix (str(response) — returns Python list repr):
# Responses containing Python list repr:  50 / 50 (100%)
# Responses starting with "[{":          50 / 50 (100%)
# Clean text responses:                   0 / 50 (0%)
# Duration: 3 weeks July–October 2026
# Users affected: 500 daily active users (internal knowledge base assistant)

# After fix (response[0]["generated_text"]):
# Clean text responses:                  50 / 50 (100%)
# Responses starting with "[{":          0 / 50 (0%)
# Average response length:               312 characters
# Average endpoint latency (p50):        3.2s (ml.g5.2xlarge, Llama 3.1 8B)
# Average endpoint latency (p95):        5.8s

# Additional check: return_full_text=False added to parameters
# Prompt echo in response:               0 / 50 (0%)

# Production result (Northbrook ML Systems Ltd.):
# User-visible list repr in responses:  100% → 0%
# Slack bot response format: clean text (no more Python syntax visible to users)
# 1.5h retainer engagement.
# Triage: 0.3h — add type(response) logging; confirm response is list not string.
# Root cause: 0.3h — confirm TGI list response format; identify str() anti-pattern.
# Fix: 0.3h — replace str(response) with response[0]['generated_text']; add assertions.
# Validation: 0.6h — 50-request batch; confirm all responses clean; add CI test; 1.5h total.

Related retainer posts

Track retainer hours without a client portal

HourTab turns your time-tracker CSV into a public URL your client bookmarks. No login. No portal. CSV in, URL out.

See pricing →