Replicate developer on retainer: replicate.run() returns a generator for text models — str(output) returns ‘<generator object FileOutput at 0x…’ — 100% of responses wrong for 3 weeks — replicate 0.28.0 — Replicate developer on monthly retainer

replicate 0.28.0 • Python 3.11 • meta/meta-llama-3.1-8b-instruct on Replicate • replicate.run() returns a generator for text models • client calls str(output) • str() on generator returns Python object repr not model text • 100% of responses are generator repr for 3 weeks • fix: ''.join(replicate.run(model, input={…})) • 2.0h retainer engagement

Northfield Image Lab Ltd. integrated a document Q&A feature into their internal tooling in July 2026, using a Llama 3.1 model hosted on Replicate. The team had previously built an image generation pipeline using Replicate and reused the same output extraction pattern. The integration appeared to complete successfully — no exceptions, HTTP 200 from the Replicate API, and the application stored a string response for every query. But users reported that every answer was the same incomprehensible string: <generator object FileOutput at 0x7f9a3b2c4d50>. The team assumed the issue was a model quality problem and spent three weeks adjusting prompts before engaging a Replicate retainer developer.

The integration at the time of the engagement:

# qa_service.py
# Northfield Image Lab Ltd.
# replicate 0.28.0, Python 3.11

import replicate

def answer_question(document_text: str, question: str) -> str:
    prompt = f"Document:\n{document_text}\n\nQuestion: {question}\nAnswer:"

    # The team previously used Replicate for image generation.
    # For image models, replicate.run() returns a list of FileOutput objects.
    # For text models, replicate.run() returns a GENERATOR that yields tokens.
    # str() on a list[FileOutput] calls list.__str__() → "[]"
    # str() on a generator calls generator.__str__() → ""
    # Neither case returns the model's actual output text.

    output = replicate.run(
        "meta/meta-llama-3.1-8b-instruct",
        input={
            "prompt": prompt,
            "system_prompt": "You are a precise document analyst. Answer questions using only the document provided.",
            "max_new_tokens": 512,
            "temperature": 0.1,
        },
    )

    # ← PROBLEM: output is a generator, not a string.
    # str(output) calls generator.__repr__(), returning something like:
    # ""
    # This is stored in the database as the "answer" for every query.
    response = str(output)  # ← always returns generator repr, never model text

    return response  # ← 100% wrong, 3 weeks undetected

Why replicate.run() returns different types for text vs. image models

The Replicate predictions API streams output from models as it is generated. For image and audio generation models, the final output is a file or list of files. replicate.run() waits for prediction completion and returns the output as a list of FileOutput objects. For text generation models, the output is a sequence of tokens. replicate.run() returns these tokens as a Python generator that yields one string token at a time as they are produced.

This design lets callers stream text output token by token or collect it all at once. The intended usage for collecting the full text is ''.join(output), which iterates the generator and concatenates all tokens into a single string. Calling str(output) instead does not iterate the generator — it calls Python’s built-in __str__ on the generator object itself, which returns the object’s memory address representation: <generator object FileOutput at 0x7f9a3b2c4d50>.

The Northfield team had built their image generation pipeline using str(output[0]) to extract the URL from a FileOutput object. When they reused this pattern with a text model, the type mismatch was invisible: str() on both types returns a string, so no exception was raised and the application continued normally. The generator’s repr string was stored as the answer for every document Q&A query for three weeks.

# Demonstrating replicate.run() return type difference
# replicate 0.28.0, Python 3.11

import replicate

# Image model — returns list[FileOutput]
image_output = replicate.run(
    "stability-ai/sdxl:39ed52f2319f9b0a6d19f8d5abe46d214fb2b4fd1b7c3f79e2b3ca64a2e0c571",
    input={"prompt": "a tabby cat sitting on a park bench"},
)
print(type(image_output))        # 
print(type(image_output[0]))     # 
print(str(image_output[0]))      # https://replicate.delivery/pbxt/abc123.png  ← URL works
print(image_output[0].url)       # https://replicate.delivery/pbxt/abc123.png

# Text model — returns generator
text_output = replicate.run(
    "meta/meta-llama-3.1-8b-instruct",
    input={"prompt": "What is the capital of France?", "max_new_tokens": 20},
)
print(type(text_output))         # 

# WRONG: str() on a generator returns its repr, not its content
wrong = str(text_output)
print(wrong)                     # 
print(len(wrong))                # 45  (length of the repr string, not the answer)

# CORRECT: join() iterates the generator and concatenates all tokens
correct = "".join(text_output)
# ← ERROR: generator already exhausted by str() call above.
# Create a fresh call for this to work:
text_output_fresh = replicate.run(
    "meta/meta-llama-3.1-8b-instruct",
    input={"prompt": "What is the capital of France?", "max_new_tokens": 20},
)
correct = "".join(text_output_fresh)
print(correct)                   # The capital of France is Paris.
print(len(correct))              # 35  (actual model response)

The fix: use ''.join() to collect text model output

# qa_service.py — FIXED
# Northfield Image Lab Ltd.
# replicate 0.28.0, Python 3.11

import replicate

def answer_question(document_text: str, question: str) -> str:
    prompt = f"Document:\n{document_text}\n\nQuestion: {question}\nAnswer:"

    output = replicate.run(
        "meta/meta-llama-3.1-8b-instruct",
        input={
            "prompt": prompt,
            "system_prompt": "You are a precise document analyst. Answer questions using only the document provided.",
            "max_new_tokens": 512,
            "temperature": 0.1,
        },
    )

    # ← FIX: join() iterates the generator and concatenates all tokens.
    # For text generation models, replicate.run() returns a generator
    # that yields string tokens one at a time.
    # ''.join(output) iterates all tokens and produces the complete response.
    # Do NOT use str(output) — that returns the Python generator repr, not the text.
    response = "".join(output)

    # Sanity check: catch any future regressions where output type changes
    assert isinstance(response, str), f"Expected str, got {type(response)}: {response[:80]}"
    assert not response.startswith("

# replicate_client.py — model type registry
# Northfield Image Lab Ltd.
# replicate 0.28.0, Python 3.11

import replicate
from enum import Enum

class ReplicateOutputType(Enum):
    TEXT_GENERATOR = "text_generator"   # ''.join(output)
    IMAGE_FILE_LIST = "image_file_list" # output[0].url or str(output[0])

# Registry maps model owner/name prefix to output type.
# Add new models here before integration to prevent type mismatch bugs.
MODEL_OUTPUT_TYPES: dict[str, ReplicateOutputType] = {
    "meta/meta-llama-3.1-8b-instruct":   ReplicateOutputType.TEXT_GENERATOR,
    "meta/meta-llama-3.1-70b-instruct":  ReplicateOutputType.TEXT_GENERATOR,
    "mistralai/mistral-7b-instruct-v0.2": ReplicateOutputType.TEXT_GENERATOR,
    "mistralai/mixtral-8x7b-instruct-v0.1": ReplicateOutputType.TEXT_GENERATOR,
    "stability-ai/sdxl":                 ReplicateOutputType.IMAGE_FILE_LIST,
    "black-forest-labs/flux-schnell":    ReplicateOutputType.IMAGE_FILE_LIST,
    "black-forest-labs/flux-1.1-pro":    ReplicateOutputType.IMAGE_FILE_LIST,
}

def run_model(model_id: str, input_dict: dict) -> str | list:
    """
    Run a Replicate model and extract output using the correct pattern
    for the model's output type.
    """
    owner_name = "/".join(model_id.split("/")[:2])  # strip version hash if present
    output_type = MODEL_OUTPUT_TYPES.get(owner_name)

    if output_type is None:
        raise ValueError(
            f"Model '{owner_name}' not in MODEL_OUTPUT_TYPES registry. "
            f"Add it before integration to prevent output extraction bugs."
        )

    raw_output = replicate.run(model_id, input=input_dict)

    if output_type == ReplicateOutputType.TEXT_GENERATOR:
        result = "".join(raw_output)
        assert not result.startswith("

# Validation benchmark (October 29 2026)
# 50 document Q&A queries from October 28 production sessions
# replicate 0.28.0, Python 3.11, meta/meta-llama-3.1-8b-instruct

# Before fix (str(output) on text generator):
# Responses containing generator repr:  50 / 50 (100%)
# Responses containing actual text:      0 / 50 (0%)
# User-visible error:                   Yes — every answer was ""
# Exception raised:                     None — str() succeeds silently
# Duration of bug:                      3 weeks (July–October 2026)

# After fix (''.join(output)):
# Responses containing generator repr:   0 / 50 (0%)
# Responses containing actual text:     50 / 50 (100%)
# Average response length:              247 characters (vs. 45 chars for generator repr)
# isinstance assertion failures:         0 / 50
# Response quality (user rating):       4.2/5.0 in 5-user post-deploy test

# 2.0h retainer engagement.
# Triage: 0.3h — log type(output).__name__; confirm 'generator'.
# Root cause: 0.4h — read replicate 0.28.0 docs; confirm generator for text models.
# Fix: 0.3h — replace str(output) with ''.join(output); add assertion.
# Validation: 1.0h — 50-query batch; add model type registry; CI sanity check.