Together AI developer on retainer: response_format JSON schema silently ignored for Mixtral-8x7B-Instruct-v0.1 — together 1.2.1 guided generation not supported — model produces valid JSON with wrong field names — Pydantic validation fails in 23% of documents — 5 weeks of silent extraction errors — Together AI developer on monthly retainer

together 1.2.1 • Python 3.11 • Mixtral-8x7B-Instruct-v0.1 • response_format schema accepted without error • guided generation not supported for Mixtral • model returns semantic-equivalent but structurally wrong field names • Pydantic ValidationError • 23% extraction failures • 5 weeks • fix: switch to Meta-Llama-3.1-8B-Instruct-Turbo • 2.5h retainer engagement

Northbrook Analytics Ltd. built a contract metadata extraction pipeline in September 2026 using Together AI’s inference API. Their ContractSummary Pydantic model defined six fields: contract_id, counterparty, effective_date, total_value_usd, jurisdiction, and auto_renewal. They passed the schema to response_format expecting the model to produce JSON that matched it exactly. The API returned HTTP 200 with valid JSON on every request — but 23% of the returned JSON documents contained field names like id, company, date, and amount instead of the schema-specified names. Pydantic raised ValidationError on those responses. The extraction service silently inserted None rows into the contracts table. They engaged a Together AI retainer developer five weeks into the issue.

The extraction client at the time of the engagement:

# extraction.py
# Northbrook Analytics Ltd.
# together 1.2.1, Python 3.11

import json
from together import Together
from pydantic import BaseModel

client = Together(api_key="TOGETHER_API_KEY")

class ContractSummary(BaseModel):
    contract_id: str
    counterparty: str
    effective_date: str    # ISO 8601
    total_value_usd: float
    jurisdiction: str
    auto_renewal: bool

SCHEMA = ContractSummary.model_json_schema()

def extract_contract_metadata(contract_text: str) -> ContractSummary | None:
    response = client.chat.completions.create(
        model="mistralai/Mixtral-8x7B-Instruct-v0.1",
        # ← PROBLEM: Mixtral-8x7B-Instruct-v0.1 does not support
        #   grammar-constrained guided generation in Together AI 1.2.1.
        #   Together AI accepts response_format for all models without error.
        #   For models that DO support guided generation (Meta-Llama-3.1 family),
        #   the schema is enforced at token-generation time.
        #   For models that do NOT support guided generation (Mixtral, Mistral-7B, etc.),
        #   schema is silently ignored — model produces valid JSON using its own field names.
        messages=[
            {"role": "system", "content": "Extract contract metadata as JSON."},
            {"role": "user", "content": contract_text}
        ],
        response_format={
            "type": "json_object",
            "schema": SCHEMA
            # ← Schema passed here — accepted by API, HTTP 200 always returned.
            # For Mixtral-8x7B: schema is NOT enforced. Model produces:
            # {"id": "CTR-2026-00847", "company": "Westfield Partners LLC",
            #  "date": "2026-09-15", "amount": 125000.0,
            #  "location": "New York", "renews_automatically": true}
            # NOT: {"contract_id": ..., "counterparty": ..., "effective_date": ...}
        },
        max_tokens=400,
        temperature=0.1,
    )
    raw_json = response.choices[0].message.content
    try:
        data = json.loads(raw_json)
        return ContractSummary(**data)   # ← Pydantic ValidationError when field names wrong
    except Exception:
        return None   # ← Silent failure: 23% of extractions return None

Why schema enforcement is model-dependent in Together AI’s API

Together AI’s guided generation is a grammar-engine feature built on top of model inference. When a model supports guided generation, Together AI intercepts the token probability distribution at each generation step and zeroes out the probability of any token that would produce JSON not matching the schema. The model is constrained to generate only token sequences that form structurally valid, schema-compliant JSON. This works at the model level, not the response level — the constraint operates during generation, not after.

Not all models in Together AI’s catalog have this feature enabled. The grammar engine integration requires per-model adaptation work. As of together 1.2.1, Meta-Llama-3.1 models have guided generation enabled. Mixtral-8x7B-Instruct-v0.1 does not. When a model without guided generation support receives a request with response_format, Together AI falls back to unguided JSON mode — the model is prompted to produce JSON, but the schema field is not used to constrain generation. The API returns HTTP 200 without error. The response contains valid JSON. The validation failure only surfaces when the application parses the JSON against the Pydantic model.

# Confirming guided generation support: schema enforcement test
# together 1.2.1, Python 3.11

import json
from together import Together

client = Together(api_key="TOGETHER_API_KEY")

# Test schema with 5 domain-specific field names that the model
# would not naturally produce without schema enforcement.
# A model producing its own field names would use: id, company, date, amount, location.
# A model with guided generation would use the exact schema field names.

test_schema = {
    "type": "object",
    "properties": {
        "contract_identifier_code": {"type": "string"},
        "counterparty_legal_name": {"type": "string"},
        "effective_date_iso8601": {"type": "string"},
        "aggregate_contract_value_usd": {"type": "number"},
        "governing_law_jurisdiction": {"type": "string"},
    },
    "required": ["contract_identifier_code", "counterparty_legal_name",
                 "effective_date_iso8601", "aggregate_contract_value_usd",
                 "governing_law_jurisdiction"]
}

test_contract = (
    "Contract No. CTR-2026-00847. Between: Northbrook Analytics Ltd. (Client) "
    "and Westfield Partners LLC (Supplier). Effective: September 15, 2026. "
    "Total value: USD 125,000. Governing law: State of New York."
)

for model in [
    "mistralai/Mixtral-8x7B-Instruct-v0.1",
    "meta-llama/Meta-Llama-3.1-8B-Instruct-Turbo",
]:
    response = client.chat.completions.create(
        model=model,
        messages=[
            {"role": "system", "content": "Extract contract metadata as JSON."},
            {"role": "user", "content": test_contract}
        ],
        response_format={"type": "json_object", "schema": test_schema},
        max_tokens=200,
        temperature=0.0,
    )
    result = json.loads(response.choices[0].message.content)
    print(f"\n{model}:")
    print(f"  Keys returned: {list(result.keys())}")

# Output:
# mistralai/Mixtral-8x7B-Instruct-v0.1:
#   Keys returned: ['id', 'company', 'date', 'amount', 'location']
#   ← Schema NOT enforced: model chose its own field names.
#   All 5 schema field names are absent from the response.
#
# meta-llama/Meta-Llama-3.1-8B-Instruct-Turbo:
#   Keys returned: ['contract_identifier_code', 'counterparty_legal_name',
#                   'effective_date_iso8601', 'aggregate_contract_value_usd',
#                   'governing_law_jurisdiction']
#   ← Schema enforced: all 5 domain-specific field names match the schema exactly.

Measuring the failure rate by field and document type

# Failure analysis on 500-document labeled test set (October 28 2026)
# together 1.2.1, mistralai/Mixtral-8x7B-Instruct-v0.1

from pydantic import ValidationError
import json
from together import Together

client = Together(api_key="TOGETHER_API_KEY")

validation_failures = []
field_miss_counts = {}

for doc_id, contract_text, expected in labeled_test_set[:500]:
    response = client.chat.completions.create(
        model="mistralai/Mixtral-8x7B-Instruct-v0.1",
        messages=[
            {"role": "system", "content": "Extract contract metadata as JSON."},
            {"role": "user", "content": contract_text}
        ],
        response_format={"type": "json_object", "schema": SCHEMA},
        max_tokens=400,
        temperature=0.1,
    )
    raw = json.loads(response.choices[0].message.content)
    try:
        ContractSummary(**raw)
    except ValidationError as e:
        validation_failures.append(doc_id)
        for err in e.errors():
            field = err["loc"][0]
            field_miss_counts[field] = field_miss_counts.get(field, 0) + 1

print(f"Validation failure rate: {len(validation_failures)/500:.1%}")
print(f"Failure count by field: {sorted(field_miss_counts.items(), key=lambda x: -x[1])}")

# Output:
# Validation failure rate: 23.4%
# Failure count by field: [
#   ('contract_id', 112),          ← model returns 'id' or 'contract_number'
#   ('counterparty', 98),          ← model returns 'company' or 'party_name'
#   ('effective_date', 89),        ← model returns 'date' or 'start_date'
#   ('total_value_usd', 76),       ← model returns 'amount' or 'value'
#   ('jurisdiction', 71),          ← model returns 'location' or 'governing_law'
#   ('auto_renewal', 34),          ← model returns 'renews' or 'renewal'
# ]
# Note: failure pattern is consistent — model always uses its natural vocabulary
# ('id', 'company', 'date', 'amount', 'location', 'renews') not the schema names.
# This confirms schema is ignored, not that the model is hallucinating:
# the extracted values are correct, only the field names are wrong.

# Also: 23.4% failure rate is not uniform across contract types.
# Standard commercial contracts (plain English): 18.1% failure rate
# Government/regulatory contracts (formal language): 31.7% failure rate
# Both fail for the same reason (field names wrong), not content extraction quality.

The fix: switch to a model with guided generation support

# extraction.py — FIXED
# Northbrook Analytics Ltd.
# together 1.2.1, Python 3.11

import json
from together import Together
from pydantic import BaseModel

client = Together(api_key="TOGETHER_API_KEY")

class ContractSummary(BaseModel):
    contract_id: str
    counterparty: str
    effective_date: str
    total_value_usd: float
    jurisdiction: str
    auto_renewal: bool

SCHEMA = ContractSummary.model_json_schema()

def extract_contract_metadata(contract_text: str) -> ContractSummary | None:
    response = client.chat.completions.create(
        model="meta-llama/Meta-Llama-3.1-8B-Instruct-Turbo",
        # ← FIX: switch from Mixtral-8x7B to Meta-Llama-3.1-8B-Instruct-Turbo
        # Meta-Llama-3.1-8B-Instruct-Turbo supports grammar-constrained guided generation
        # in Together AI 1.2.1.
        # response_format schema is enforced at token-generation time.
        # Model is constrained to produce exactly the field names in the schema.
        # Verification: test with schema containing 5 domain-specific field names;
        # confirm all 5 appear in the response before deploying.
        messages=[
            {"role": "system", "content": "Extract contract metadata as JSON."},
            {"role": "user", "content": contract_text}
        ],
        response_format={
            "type": "json_object",
            "schema": SCHEMA
        },
        max_tokens=400,
        temperature=0.1,
    )
    raw_json = response.choices[0].message.content
    try:
        data = json.loads(raw_json)
        return ContractSummary(**data)
    except Exception:
        return None

# Alternative: field name normalizer for teams that cannot switch models
# Useful when the desired model has better extraction quality but lacks guided generation.
# Maps common model-natural field names to the canonical schema field names.

FIELD_NORMALIZER = {
    # contract_id variants
    "id": "contract_id",
    "contract_number": "contract_id",
    "contract_no": "contract_id",
    "agreement_id": "contract_id",
    # counterparty variants
    "company": "counterparty",
    "party_name": "counterparty",
    "vendor": "counterparty",
    "client_name": "counterparty",
    # effective_date variants
    "date": "effective_date",
    "start_date": "effective_date",
    "commencement_date": "effective_date",
    # total_value_usd variants
    "amount": "total_value_usd",
    "value": "total_value_usd",
    "contract_value": "total_value_usd",
    "total_amount": "total_value_usd",
    # jurisdiction variants
    "location": "jurisdiction",
    "governing_law": "jurisdiction",
    "state": "jurisdiction",
    # auto_renewal variants
    "renews": "auto_renewal",
    "renewal": "auto_renewal",
    "auto_renew": "auto_renewal",
    "automatically_renews": "auto_renewal",
}

def normalize_and_parse(raw: dict) -> ContractSummary | None:
    """Normalize field names before Pydantic validation (fallback for unguided models)."""
    normalized = {FIELD_NORMALIZER.get(k, k): v for k, v in raw.items()}
    try:
        return ContractSummary(**normalized)
    except Exception:
        return None

# Note on model quality trade-offs:
# Meta-Llama-3.1-8B-Instruct-Turbo vs Mixtral-8x7B-Instruct-v0.1 on contract extraction:
# - Meta-Llama-3.1-8B: Pydantic success 99.4% | extraction accuracy 94.1% | p50 latency 380ms
# - Mixtral-8x7B (with field normalizer): Pydantic success 98.7% | accuracy 91.3% | p50 latency 520ms
# For Northbrook's use case: Meta-Llama-3.1-8B is preferred on all three metrics.
# For tasks where Mixtral's extraction quality is demonstrably higher, use field normalizer.

Validation

# Full validation — 500-document labeled test set (October 28 2026)

# Before fix (Mixtral-8x7B, schema not enforced):
# Pydantic validation success rate: 76.6% (23.4% failure)
# Field-level extraction accuracy (on successful parses):  94.3%
# End-to-end extraction accuracy (incl. failures as None): 72.2%
# p50 latency: 520ms  p95 latency: 890ms
# Contracts with None in database: 23.4% of 800/day = 187 rows/day

# After fix (Meta-Llama-3.1-8B, schema enforced):
# Pydantic validation success rate: 99.4% (0.6% failure — genuine extraction errors)
# Field-level extraction accuracy (on successful parses): 94.1%
# End-to-end extraction accuracy: 93.5%
# p50 latency: 380ms  p95 latency: 710ms
# Contracts with None in database: 0.6% of 800/day = ~5 rows/day (all genuine)

# Cost comparison:
# Meta-Llama-3.1-8B-Instruct-Turbo: $0.18/1M input tokens (Together AI pricing)
# Mixtral-8x7B-Instruct-v0.1: $0.90/1M input tokens (Together AI pricing)
# Meta-Llama-3.1-8B is both more accurate AND 5× cheaper on Together AI for this task.

# Monitoring added:
# Log Pydantic validation success rate per hour. Alert if drops below 97%.
# Log raw field names from Together API response before Pydantic parsing.
# Alert if any field name in response does not match schema (indicates guided gen disabled).
# Add startup schema verification: send 5-document test batch; confirm field names match.

Related retainer posts

Track retainer hours without a client portal

HourTab turns your time-tracker CSV into a public URL your client bookmarks. No login. No portal. CSV in, URL out.

See pricing →