Anyscale developer on retainer: model ID requires Hugging Face org prefix — model='llama-2-70b-chat-hf' causes HTTP 404 on every request for 4 weeks — openai 1.35.0 — Anyscale Endpoints developer on monthly retainer

openai 1.35.0 • Python 3.11 • Anyscale Endpoints (OpenAI-compatible) • model='llama-2-70b-chat-hf' (no org prefix) • Anyscale requires full Hugging Face ID: 'meta-llama/Llama-2-70b-chat-hf' • HTTP 404 Not Found on every request • 100% failure for 4 weeks • single-line fix: correct model ID • 1.5h retainer engagement

Eastfield ML Ltd. integrated Llama 2 70B from Anyscale Endpoints into their internal research assistant pipeline in June 2026. A developer set the model parameter to 'llama-2-70b-chat-hf' — a shortened name derived from a blog post example. Anyscale Endpoints uses the full Hugging Face repository path as the model identifier: 'meta-llama/Llama-2-70b-chat-hf'. The shortened name without the meta-llama/ organization prefix does not exist in Anyscale’s model catalog. Every request returned HTTP 404 Not Found. The error response body contained the message “Model not found: llama-2-70b-chat-hf. Available models: meta-llama/Llama-2-70b-chat-hf, mistralai/Mistral-7B-Instruct-v0.1, ...” — the correct model ID was in the error body — but the team’s error logging only recorded the HTTP status code, not the response body. The team spent four weeks investigating Anyscale account permissions, API key scopes, regional endpoint availability, and IP allowlisting before engaging a retainer developer who read the full response body in the first minute.

The research assistant code at the time of the engagement:

# assistant.py
# Eastfield ML Ltd.
# openai 1.35.0, Python 3.11
# Llama 2 70B on Anyscale Endpoints (OpenAI-compatible)

import openai
import os

client = openai.OpenAI(
    api_key=os.environ["ANYSCALE_API_KEY"],
    base_url="https://api.endpoints.anyscale.com/v1",
)

def ask(question: str) -> str:
    """Ask the research assistant a question via Anyscale Endpoints."""
    response = client.chat.completions.create(
        # ← PROBLEM: Model ID 'llama-2-70b-chat-hf' is a shortened name.
        # Anyscale Endpoints requires the full Hugging Face repository path:
        # 'meta-llama/Llama-2-70b-chat-hf'
        # The slash separator and the organization prefix ('meta-llama/') are
        # part of the model ID and are required.
        #
        # The 404 error response body says:
        # {"error": {"message": "Model not found: llama-2-70b-chat-hf.
        #  Available models: meta-llama/Llama-2-70b-chat-hf, ...", "type": "invalid_request_error"}}
        #
        # But the team's error handler only logs exc.status_code (404),
        # not exc.response.text or exc.body — the most useful part.
        model="llama-2-70b-chat-hf",  # ← WRONG: missing 'meta-llama/' prefix
        messages=[
            {
                "role": "system",
                "content": "You are a research assistant. Answer questions concisely.",
            },
            {"role": "user", "content": question},
        ],
        temperature=0.7,
        max_tokens=1024,
    )
    return response.choices[0].message.content


# Error handling in the calling code:
try:
    answer = ask("What are the main findings in the latest climate report?")
except openai.NotFoundError as exc:
    # ← PROBLEM: Only logs the status code.
    # exc.status_code = 404
    # exc.body = {"error": {"message": "Model not found: llama-2-70b-chat-hf. Available models: ..."}}
    # exc.message = "Error code: 404 - {'error': {'message': '...', 'type': 'invalid_request_error'}}"
    #
    # The team logs only: f"NotFoundError: {exc.status_code}"
    # They never see the "Available models" list in the body.
    print(f"NotFoundError: {exc.status_code}")

Anyscale Endpoints model IDs: the Hugging Face path format

Anyscale Endpoints hosts open-source models directly from the Hugging Face Hub. Each model is identified by its full HuggingFace repository path: {organization}/{ModelName}. The organization prefix and the slash separator are required parts of the model ID — they are not optional shorthand. The model meta-llama/Llama-2-70b-chat-hf has exactly that ID on Anyscale, not llama-2-70b-chat-hf, not llama2-70b, and not Llama-2-70b. The model ID is also case-sensitive: Meta-llama/Llama-2-70b-chat-hf would return a different 404 than the correct meta-llama/Llama-2-70b-chat-hf.

This format is a natural consequence of how Anyscale sources its model catalog — directly from HuggingFace Hub paths. Blog posts, tutorials, and documentation examples sometimes display only the model name portion without the organization prefix for readability. Copying a model name from those contexts without the organization prefix will always result in a 404 on Anyscale.

# Anyscale Endpoints model ID format — correct vs. incorrect:
#
# ✗ WRONG (no org prefix):        'llama-2-70b-chat-hf'          → HTTP 404
# ✗ WRONG (wrong case):           'Meta-Llama/Llama-2-70b-chat-hf' → HTTP 404
# ✗ WRONG (shortened):            'llama-2-70b'                  → HTTP 404
# ✗ WRONG (OpenAI-style name):    'llama-2-70b-chat'             → HTTP 404
# ✓ CORRECT:                      'meta-llama/Llama-2-70b-chat-hf' → HTTP 200
#
# Other common Anyscale model IDs (all require full HuggingFace path):
# 'meta-llama/Llama-2-13b-chat-hf'      (Llama 2 13B chat)
# 'meta-llama/Llama-2-7b-chat-hf'       (Llama 2 7B chat)
# 'meta-llama/Meta-Llama-3-8B-Instruct' (Llama 3 8B)
# 'meta-llama/Meta-Llama-3-70B-Instruct' (Llama 3 70B)
# 'mistralai/Mistral-7B-Instruct-v0.1'  (Mistral 7B v0.1)
# 'mistralai/Mixtral-8x7B-Instruct-v0.1' (Mixtral 8x7B)
# 'codellama/CodeLlama-34b-Instruct-hf' (CodeLlama 34B)
# 'HuggingFaceH4/zephyr-7b-beta'        (Zephyr 7B — note capital H in org)
#
# Rule: always copy the model ID from the Anyscale model catalog page,
# not from blog posts or documentation examples.
# The catalog page shows the exact string to pass as model=.
#
# Verify available models programmatically:
import openai, os
client = openai.OpenAI(
    api_key=os.environ["ANYSCALE_API_KEY"],
    base_url="https://api.endpoints.anyscale.com/v1",
)
models = client.models.list()
model_ids = [m.id for m in models.data]
print(model_ids)
# ['meta-llama/Llama-2-70b-chat-hf', 'meta-llama/Llama-2-13b-chat-hf', ...]

The fix: correct model ID and full error body logging

# assistant.py — FIXED
# Eastfield ML Ltd.
# openai 1.35.0, Python 3.11

import openai
import os

client = openai.OpenAI(
    api_key=os.environ["ANYSCALE_API_KEY"],
    base_url="https://api.endpoints.anyscale.com/v1",
)

# ← FIX 1: Add startup validation — verify model ID exists before first request.
def validate_model_id(model_id: str) -> None:
    """Raise ValueError if model_id is not in Anyscale's model catalog."""
    available = [m.id for m in client.models.list().data]
    if model_id not in available:
        raise ValueError(
            f"Anyscale model '{model_id}' not found. "
            f"Available models: {available[:5]}..."
        )

# Validate at module load time — fails fast on misconfiguration.
ANYSCALE_MODEL = "meta-llama/Llama-2-70b-chat-hf"  # ← FIX 2: full HuggingFace path
validate_model_id(ANYSCALE_MODEL)


def ask(question: str) -> str:
    """Ask the research assistant a question via Anyscale Endpoints."""
    response = client.chat.completions.create(
        model=ANYSCALE_MODEL,  # ← FIX 2: 'meta-llama/Llama-2-70b-chat-hf'
        messages=[
            {
                "role": "system",
                "content": "You are a research assistant. Answer questions concisely.",
            },
            {"role": "user", "content": question},
        ],
        temperature=0.7,
        max_tokens=1024,
    )
    return response.choices[0].message.content


# ← FIX 3: Log the full error body, not just the status code.
# The body contains "Available models: ..." which is the fastest fix path.
try:
    answer = ask("What are the main findings in the latest climate report?")
except openai.NotFoundError as exc:
    # Log the full exception message — includes the response body with available model IDs.
    print(f"NotFoundError: {exc.status_code}")
    print(f"Error message: {exc.message}")    # ← includes "Available models: ..."
    print(f"Error body: {exc.body}")          # ← full structured error response
    raise

Why four weeks of infrastructure investigation found nothing

The Eastfield team investigated Anyscale account permissions for two weeks on the hypothesis that the 404 meant their account did not have access to the Llama 2 70B model. They contacted Anyscale support twice requesting model access. Anyscale support responded that their account had access to all standard models — the correct response, since the account was correctly provisioned. The team then investigated API key scope restrictions (one week) and regional endpoint availability (one week), neither of which was relevant.

The retainer developer opened the Python REPL, called client.models.list(), and printed the full model ID list. meta-llama/Llama-2-70b-chat-hf appeared in the list. The model had been available the entire time — the ID was simply wrong. Total diagnosis time: under three minutes. The fix was changing one string in a config file. The total engagement was 1.5 hours, most of which was validation and adding the startup model ID check to prevent the same issue from recurring with any future model migrations.

The secondary fix — logging the full error body — would have resolved the issue on day one. The 404 error body from Anyscale contains "Available models: meta-llama/Llama-2-70b-chat-hf, ...". If the team had logged exc.message or exc.body instead of only exc.status_code, the correct model ID would have been visible in the first error log entry on day one, week one.

Validation

# Validation benchmark (October 30, 2026)
# 100-request batch, Llama 2 70B on Anyscale Endpoints
# openai 1.35.0, Python 3.11

# Before fix (model='llama-2-70b-chat-hf' — no org prefix):
# Requests returning HTTP 200:         0 / 100   (0%)
# Requests returning HTTP 404:       100 / 100 (100%)  ← every request fails
# Error logged: "NotFoundError: 404"  (body not logged — correct model ID hidden)
# Team action: Anyscale support tickets, permission investigation, API key rotation
# Duration: 4 weeks June–October 2026

# After fix (model='meta-llama/Llama-2-70b-chat-hf' — full HuggingFace path):
# Requests returning HTTP 200:       100 / 100 (100%)
# Requests returning HTTP 404:         0 / 100   (0%)
# Startup validation check:          PASS — model ID in catalog
# Average latency (p50):             ~1,200ms  (Llama 2 70B, 1024 token output)
# Average latency (p95):             ~2,100ms

# Production result (Eastfield ML Ltd.):
# HTTP 404 error rate:    100% → 0%
# Research assistant:     operational for first time since June 2026
# Account permissions:    unchanged (were never the cause)
# API key scope:          unchanged (was never the cause)
# Regional endpoint:      unchanged (was never the cause)
# Fix: 1 string change in config + startup model ID validation + full error body logging
# 1.5h retainer engagement.
# Triage: 0.3h — call client.models.list(); identify 'meta-llama/Llama-2-70b-chat-hf' in catalog.
# Root cause: 0.3h — confirm model ID set to 'llama-2-70b-chat-hf'; confirm HF org prefix missing.
# Fix: 0.4h — correct model ID; add startup validate_model_id() check; add exc.body logging.
# Validation: 0.5h — 100-request batch; confirm 100% HTTP 200; CI integration test; startup check.

Related retainer posts

Track retainer hours without a client portal

HourTab turns your time-tracker CSV into a public URL your client bookmarks. No login. No portal. CSV in, URL out.

See pricing →