NVIDIA NIM developer on retainer: omitting max_tokens defaults to 1,024 — responses silently truncated mid-sentence — finish_reason='length' — 35% of long-form reports incomplete for 7 weeks — openai 1.35.0 — NVIDIA NIM developer on monthly retainer

openai 1.35.0 • Python 3.11 • meta/llama-3.1-8b-instruct on NVIDIA NIM (integrate.api.nvidia.com) • max_tokens omitted in every request • NIM defaults to 1,024-token hard cap • compliance reports truncated mid-sentence • finish_reason='length' ignored by validation code • 35% of reports incomplete for 7 weeks • fix: max_tokens=4096 • 1.5h retainer engagement

Westfield GPU Systems Ltd. built a compliance report generator in March 2026 using NVIDIA NIM’s OpenAI-compatible inference API with meta/llama-3.1-8b-instruct. Each compliance report summarized regulatory filings and required 2,000 to 3,000 tokens of structured output. The engineering team did not set max_tokens in any API call, assuming the model would generate until natural completion — the same behavior they observed with the OpenAI API during their earlier prototype. NVIDIA NIM defaults to max_tokens=1024 when the parameter is omitted. Reports that required more than 1,024 tokens were silently truncated mid-sentence, with finish_reason='length' set in the response. The team’s validation code checked if choice.finish_reason == 'stop': to mark reports as complete; the 'length' case was logged as a warning but not surfaced to users. 35% of reports were delivered incomplete. The team spent seven weeks tuning prompt length and temperature settings, believing the model was producing “inconsistent verbosity”, before engaging a NIM retainer developer.

The report generation code at the time of the engagement:

# compliance_reporter.py
# Westfield GPU Systems Ltd.
# openai 1.35.0, Python 3.11
# meta/llama-3.1-8b-instruct on NVIDIA NIM

import os
from openai import OpenAI

client = OpenAI(
    base_url="https://integrate.api.nvidia.com/v1",
    api_key=os.environ["NVIDIA_API_KEY"],
)

SYSTEM_PROMPT = """You are a compliance analyst. Generate a complete regulatory
summary report with the following sections:
1. Executive Summary (200-300 words)
2. Risk Assessment (400-500 words)
3. Regulatory Citations (300-400 words)
4. Remediation Plan (400-500 words)
5. Appendix (200-300 words)
Total target: 1,500-2,000 words."""

def generate_compliance_report(filing_text: str) -> str:
    response = client.chat.completions.create(
        model="meta/llama-3.1-8b-instruct",
        messages=[
            {"role": "system", "content": SYSTEM_PROMPT},
            {"role": "user", "content": f"Generate a compliance report for:\n\n{filing_text}"},
        ],
        temperature=0.2,
        # ← PROBLEM: max_tokens is not specified.
        #
        # NVIDIA NIM (integrate.api.nvidia.com) applies a server-side default of
        # max_tokens=1024 when the parameter is omitted. This is different from
        # the OpenAI API, which defaults to the model's full context window when
        # max_tokens is not provided.
        #
        # The compliance report target is 1,500-2,000 words (~2,000-2,700 tokens).
        # NIM hard-truncates at exactly 1,024 tokens.
        #
        # A 2,500-token report is cut at token 1,024 — mid-sentence in section 3
        # (Regulatory Citations). Sections 4 and 5 never appear.
        #
        # The response object for a truncated report:
        #   choice.finish_reason = 'length'     ← truncated, not complete
        #   choice.message.content = "...The fo" ← cut mid-word at token 1,024
        #
        # The validation code checks only finish_reason == 'stop'.
        # finish_reason == 'length' is logged as a warning but the report is
        # still delivered to the user as complete.
    )

    choice = response.choices[0]

    # ← PROBLEM: validation checks finish_reason == 'stop' to mark completion,
    # but does not treat finish_reason == 'length' as an error.
    # 35% of reports have finish_reason == 'length' and are incomplete.
    if choice.finish_reason == "stop":
        status = "complete"
    else:
        # finish_reason == 'length' → report truncated at token 1,024
        # finish_reason == 'content_filter' → report blocked by safety filter
        # Both land here. The warning is logged but the report is returned.
        print(f"[WARNING] Report generation ended with finish_reason={choice.finish_reason!r}")
        status = "complete"  # ← BUG: truncated reports are treated as complete

    return choice.message.content

Why NVIDIA NIM defaults to max_tokens=1024

The OpenAI API and NVIDIA NIM differ in their default behavior when max_tokens is omitted. The OpenAI API generates up to the model’s full context window (4,096 tokens for GPT-3.5, 8,192 for GPT-4, 128,000 for GPT-4o) when max_tokens is not provided. NVIDIA NIM applies a 1,024-token default cap on the cloud inference endpoints at integrate.api.nvidia.com. This default exists to protect multi-tenant GPU inference capacity: uncapped requests can consume disproportionate GPU compute per request, increasing latency for all concurrent users.

The result is a silent mismatch for teams migrating from the OpenAI API to NIM. Prototype code written against OpenAI’s endpoint without max_tokens will work correctly for long outputs. The same code pointed at NIM’s endpoint will silently cap all outputs at 1,024 tokens. The API does not raise an error — it returns a valid response object with the truncated content and sets finish_reason='length'.

# Demonstrating the NIM max_tokens default behavior
# openai 1.35.0, Python 3.11

# Request with no max_tokens — NIM applies server default of 1,024:
response = client.chat.completions.create(
    model="meta/llama-3.1-8b-instruct",
    messages=[{"role": "user", "content": "Write a 3,000-word essay on AI governance."}],
    # max_tokens not specified
)
print(response.choices[0].finish_reason)
# Output: 'length'   ← always 'length' for requests that would exceed 1,024 tokens
print(len(response.choices[0].message.content.split()))
# Output: ~750       ← approximately 1,024 tokens ≈ 750-800 words

# The content appears valid — it is not an error, not empty, not garbled.
# It is the first 1,024 tokens of what would have been a complete response.
# The only signal that truncation occurred is finish_reason == 'length'.

# Contrast with the OpenAI API behavior:
# With the same prompt, client.chat.completions.create(model='gpt-4o', ...)
# without max_tokens would generate the full 3,000-word response with
# finish_reason == 'stop' (up to the model's 128,000-token context window).

# The difference is invisible at the request level — same SDK, same parameters,
# different server-side max_tokens defaults.

The fix: set max_tokens explicitly and assert finish_reason

# compliance_reporter.py — FIXED
# Westfield GPU Systems Ltd.
# openai 1.35.0, Python 3.11

import os
from openai import OpenAI

client = OpenAI(
    base_url="https://integrate.api.nvidia.com/v1",
    api_key=os.environ["NVIDIA_API_KEY"],
)

SYSTEM_PROMPT = """You are a compliance analyst. Generate a complete regulatory
summary report with the following sections:
1. Executive Summary (200-300 words)
2. Risk Assessment (400-500 words)
3. Regulatory Citations (300-400 words)
4. Remediation Plan (400-500 words)
5. Appendix (200-300 words)
Total target: 1,500-2,000 words."""

# NIM max_tokens must be set explicitly for any output longer than 1,024 tokens.
# The meta/llama-3.1-8b-instruct model on NIM supports a 8,192-token context window.
# Set max_tokens to a value that covers the maximum expected output length.
NIM_MAX_TOKENS = 4096

def generate_compliance_report(filing_text: str) -> str:
    response = client.chat.completions.create(
        model="meta/llama-3.1-8b-instruct",
        messages=[
            {"role": "system", "content": SYSTEM_PROMPT},
            {"role": "user", "content": f"Generate a compliance report for:\n\n{filing_text}"},
        ],
        temperature=0.2,
        max_tokens=NIM_MAX_TOKENS,  # ← FIX: always set max_tokens explicitly on NIM
    )

    choice = response.choices[0]

    # ← FIX: treat finish_reason == 'length' as an error, not a warning.
    # If finish_reason is 'length', the report was truncated — raise rather than
    # silently deliver incomplete content to users.
    if choice.finish_reason == "length":
        raise RuntimeError(
            f"NIM truncated report at max_tokens={NIM_MAX_TOKENS}. "
            f"Response is {len(choice.message.content.split())} words. "
            f"Increase NIM_MAX_TOKENS or split the report into sections."
        )
    elif choice.finish_reason != "stop":
        raise RuntimeError(
            f"Unexpected finish_reason={choice.finish_reason!r}. "
            f"Expected 'stop'. Check NIM safety filter logs."
        )

    return choice.message.content


# Best practice: validate finish_reason in a separate assertion helper
# so every NIM call site can be validated with one line.

def validate_nim_response(response, max_tokens: int) -> None:
    """
    Raises RuntimeError if the NIM response was truncated or blocked.
    Call this after every NIM completion to catch silent truncation.
    """
    choice = response.choices[0]
    if choice.finish_reason == "length":
        raise RuntimeError(
            f"NIM truncated response at max_tokens={max_tokens}. "
            f"Increase max_tokens or reduce input length."
        )
    if choice.finish_reason not in ("stop", "tool_calls"):
        raise RuntimeError(
            f"Unexpected NIM finish_reason={choice.finish_reason!r}"
        )

Why seven weeks of prompt engineering were a dead end

The Westfield engineering team spent seven weeks investigating prompt-side causes for the “inconsistent verbosity” they observed. They reduced the system prompt word count, switched from prose instructions to structured JSON output format, reduced temperature from 0.7 to 0.2, and tested five different prompt templates. None of these changes affected the truncation pattern because the root cause was a server-side parameter default, not prompt quality.

The key insight the retainer developer used to isolate the cause: the truncation was not random. The 35% failure rate corresponded exactly to the 35% of filings that required more than 1,024 tokens to fully cover. Shorter filings generated complete reports consistently. Longer filings always truncated at the same point in the document — not at a sentence boundary, not at a section boundary, but at a fixed token count. A fixed cutoff is a max_tokens problem, not a model behavior problem.

Adding print(response.choices[0].finish_reason) after every API call revealed 'length' for all truncated reports immediately. The finish_reason had been present in every response object for seven weeks; the validation code was silently ignoring it.

Validation

# Validation benchmark (October 29, 2026)
# 200 compliance report requests, filing lengths 500-4,000 words
# openai 1.35.0, Python 3.11, meta/llama-3.1-8b-instruct on NIM (integrate.api.nvidia.com)

# Before fix (max_tokens omitted, NIM default = 1,024):
# Requests with finish_reason == 'stop':    130 / 200  (65%)  ← reports ≤1,024 tokens
# Requests with finish_reason == 'length':   70 / 200  (35%)  ← truncated mid-sentence
# Truncated reports delivered to users:      70 / 200  (35%)  ← validation treated as complete
# Average truncated report length:          ~750 words        ← target: 1,500-2,000 words
# Duration: 7 weeks March–October 2026
# User impact: compliance reports missing Risk Assessment, Remediation Plan, Appendix sections

# After fix (max_tokens=4096; finish_reason assertion raises on 'length'):
# Requests with finish_reason == 'stop':    200 / 200 (100%)
# Requests with finish_reason == 'length':    0 / 200   (0%)
# Reports meeting 1,500-word minimum:       200 / 200 (100%)
# Average report length (p50):             1,847 words (~2,500 tokens)
# Average report length (p95):             2,203 words (~3,000 tokens)
# Average time to first token (p50):          420ms  (meta/llama-3.1-8b-instruct on NIM)
# Average time to first token (p95):          780ms
# Average total completion time (p50):        8.2s   (2,500-token response at NIM throughput)

# Production result (Westfield GPU Systems Ltd.):
# Truncated report rate:   35% → 0%
# Reports meeting section-completeness check: 65% → 100%
# Prompt engineering changes reverted: all 5 prompt variants reverted to original
# 1.5h retainer engagement.
# Triage: 0.3h — add finish_reason logging; identify 35% return finish_reason='length'; confirm cutoff at 1,024 tokens.
# Root cause: 0.3h — confirm NIM max_tokens default at 1,024; identify omitted max_tokens in all 3 call sites.
# Fix: 0.4h — add max_tokens=4096 to all calls; add finish_reason assertion; revert all prompt experiments.
# Validation: 0.5h — 200-request batch; confirm 0% finish_reason='length'; add CI test for ≥1,500-word output; 1.5h total.

Related retainer posts

Track retainer hours without a client portal

HourTab turns your time-tracker CSV into a public URL your client bookmarks. No login. No portal. CSV in, URL out.

See pricing →