ChromaDB developer on retainer: embedding model switch from Sentence Transformers to OpenAI silently breaks retrieval — cosine similarity returns meaningless scores — recall drops 94% → 21% — no error raised — 3 weeks — ChromaDB developer on monthly retainer

chromadb 0.5.0 • Python 3.11 • Sentence Transformers all-MiniLM-L6-v2 (384 dims) for ingestion • OpenAI text-embedding-3-small (1536 dims) for queries • incompatible vector spaces • cosine similarity meaningless • recall@10 drops 94% → 21% • no error raised • 3 weeks • fix: explicit embedding_function + full re-ingestion • 1.5h retainer engagement

Westfield Local Vector Ltd. built a RAG-based support ticket search system in August 2026 using ChromaDB in persistent mode. The system ingested 10,000 support tickets and returned the 10 most similar tickets for any new support query. Recall@10 was 94% against a ground-truth evaluation set. In October 2026, the team decided to upgrade embedding quality and switched to OpenAI's text-embedding-3-small model for new documents and queries. Three weeks later, user complaints about poor search results prompted an investigation — recall@10 had dropped to 21%. No errors had been logged. No model calls had failed. The system was silently returning semantically irrelevant results. They engaged a ChromaDB retainer developer.

The ingestion and query code at the time of the engagement:

# rag_system.py
# Westfield Local Vector Ltd.
# chromadb 0.5.0, Python 3.11

import chromadb
from chromadb.utils.embedding_functions import OpenAIEmbeddingFunction
import os

# PersistentClient: collection survives process restarts
client = chromadb.PersistentClient(path="./chroma_data")

# ORIGINAL INGESTION (August 2026)
# No embedding_function specified → uses ChromaDB default:
#   Sentence Transformers all-MiniLM-L6-v2, 384-dimensional vectors
collection = client.get_or_create_collection(
    name="support_tickets",
    # embedding_function not specified ← BUG ROOT 1
    # ChromaDB silently uses its internal default embedding function
    # all 10,000 documents are stored as 384-dim Sentence Transformer vectors
)

# ingest_tickets.py (run once in August 2026)
# documents = [list of 10,000 ticket text strings]
# ids = [list of 10,000 ticket IDs]
collection.add(documents=documents, ids=ids)
# Each document embedded with all-MiniLM-L6-v2 → stored as 384-dim vector

# ─────────────────────────────────────────────────────────────────
# QUERY CODE (updated in October 2026 after embedding model upgrade)
# Developer switches to OpenAI embeddings for "better quality"

openai_ef = OpenAIEmbeddingFunction(
    api_key=os.environ["OPENAI_API_KEY"],
    model_name="text-embedding-3-small",  # 1536-dimensional output
)

def search_similar_tickets(query: str, n_results: int = 10) -> list[str]:
    # Generate query embedding with OpenAI (1536 dims)
    query_embedding = openai_ef([query])[0]

    # Query ChromaDB with the OpenAI embedding
    # ↓ BUG ROOT 2: stored vectors are 384-dim (Sentence Transformers)
    # ↓              query vector is 1536-dim (OpenAI text-embedding-3-small)
    # ↓ ChromaDB does not validate that embedding dimensions or models match
    # ↓ Cosine similarity computed between geometrically incompatible spaces
    # ↓ Scores are essentially random with respect to actual semantic similarity
    # ↓ No error, no warning — 10 results always returned
    results = collection.query(
        query_embeddings=[query_embedding],
        n_results=n_results,
        include=["documents", "distances"],
    )
    return results["documents"][0]

# Observed behaviour after embedding model switch:
# Query: "payment declined on checkout"
# Expected results: tickets about payment failures, declined cards, checkout errors
# Actual results: 10 random tickets with no semantic relationship to query
# Distances (cosine similarity scores): 0.42, 0.41, 0.40, ... (random range)
# Ground-truth recall@10: 94% → 21% (73 percentage point drop)
# System logs: no errors, no warnings — all queries return 10 results
# User experience: search results appear completely random

Why ChromaDB cannot detect embedding model mismatches

ChromaDB's architecture separates document storage from embedding generation. When you call collection.add(documents=[...], ids=[...]) without specifying embeddings, ChromaDB calls its configured embedding function to produce vectors and stores both the raw text and the resulting vector. When you call collection.query(query_embeddings=[emb]), ChromaDB skips the embedding function entirely and computes cosine similarity directly against the stored vectors. It has no record of which embedding model produced the stored vectors and no way to validate that the provided query embedding came from the same model.

Cosine similarity between vectors from different embedding models produces meaningless scores. Sentence Transformers all-MiniLM-L6-v2 and OpenAI text-embedding-3-small each define their own semantic geometry: the direction of a vector in one model's space carries meaning only relative to other vectors from that same model. A query vector from the OpenAI model is a point in a 1536-dimensional space where "support ticket about payment" points in a specific direction. The stored Sentence Transformer vectors are points in a geometrically unrelated 384-dimensional space. The cosine similarity between them is a number, but it does not measure semantic similarity — it measures the angle between two vectors in incompatible coordinate systems.

# Diagnosing the vector space mismatch
# Step 1: check collection metadata for embedding model record
# (Westfield had none — first sign of missing governance)
print(collection.metadata)
# Output: {}  ← no embedding_function metadata recorded at collection creation

# Step 2: inspect a stored vector to check its dimension
stored = collection.get(ids=["ticket_0001"], include=["embeddings"])
stored_vector = stored["embeddings"][0]
print(f"Stored vector dimension: {len(stored_vector)}")
# Output: Stored vector dimension: 384
# ← all-MiniLM-L6-v2 produces 384-dimensional vectors

# Step 3: generate a query embedding and check its dimension
query_embedding = openai_ef(["payment declined on checkout"])[0]
print(f"Query embedding dimension: {len(query_embedding)}")
# Output: Query embedding dimension: 1536
# ← text-embedding-3-small produces 1536-dimensional vectors

# Step 4: compute mean cosine similarity for 50 test queries
# against ground-truth top-10 results
import numpy as np

def cosine_sim(a, b):
    a, b = np.array(a), np.array(b)
    return np.dot(a, b) / (np.linalg.norm(a) * np.linalg.norm(b))

# Correct model pairing (both Sentence Transformers):
# mean top-10 cosine similarity: 0.71 (semantically related)

# Mismatch (OpenAI query, Sentence Transformer stored):
# mean top-10 cosine similarity: 0.41 (random noise range)
# ← near the expected cosine similarity for random high-dimensional vectors

# Recall@10 evaluation against 100 ground-truth query/relevant-set pairs:
# Sentence Transformers ingestion + Sentence Transformers query: 94%
# Sentence Transformers ingestion + OpenAI query:               21%
# ← 73 percentage point recall drop from embedding model mismatch

The fix: specify embedding_function explicitly and re-ingest

# rag_system_fixed.py
# chromadb 0.5.0, Python 3.11

import chromadb
from chromadb.utils.embedding_functions import OpenAIEmbeddingFunction
import os

client = chromadb.PersistentClient(path="./chroma_data")

# Always specify the embedding function explicitly
openai_ef = OpenAIEmbeddingFunction(
    api_key=os.environ["OPENAI_API_KEY"],
    model_name="text-embedding-3-small",
)

# Step 1: delete the old collection (384-dim Sentence Transformer vectors)
# and create a new one with explicit embedding_function
try:
    client.delete_collection("support_tickets")
except Exception:
    pass

collection = client.get_or_create_collection(
    name="support_tickets",
    embedding_function=openai_ef,    # ← explicit embedding function
    metadata={
        "embedding_model": "text-embedding-3-small",
        "embedding_dims": 1536,
        "embedding_provider": "openai",
        "created_at": "2026-10-31",
    },
)

# Step 2: re-ingest all 10,000 documents with the OpenAI embedding function
# Use add(documents=[...]) — ChromaDB calls openai_ef internally
BATCH_SIZE = 500
for i in range(0, len(documents), BATCH_SIZE):
    batch_docs = documents[i:i + BATCH_SIZE]
    batch_ids  = ids[i:i + BATCH_SIZE]
    collection.add(documents=batch_docs, ids=batch_ids)
    print(f"Ingested {min(i + BATCH_SIZE, len(documents))}/{len(documents)}")

# Step 3: use query_texts (not query_embeddings) — let ChromaDB call the EF
def search_similar_tickets(query: str, n_results: int = 10) -> list[str]:
    results = collection.query(
        query_texts=[query],    # ← ChromaDB calls openai_ef automatically
        n_results=n_results,
        include=["documents", "distances"],
    )
    return results["documents"][0]

# Step 4: startup assertion — validate collection metadata matches config
def assert_embedding_consistency():
    meta = collection.metadata or {}
    configured_model = "text-embedding-3-small"
    recorded_model = meta.get("embedding_model")
    if recorded_model != configured_model:
        raise RuntimeError(
            f"Embedding model mismatch: collection has '{recorded_model}', "
            f"system configured for '{configured_model}'. "
            f"Re-ingest all documents before changing embedding models."
        )

assert_embedding_consistency()  # call at application startup

# After re-ingestion:
# Stored vector dimension:  1536  (text-embedding-3-small)
# Query vector dimension:   1536  (text-embedding-3-small)
# Same model — cosine similarity semantically meaningful
# Mean top-10 cosine similarity: 0.74
# Recall@10: 21% → 94%  (73 percentage point recovery)

Why this bug is easy to introduce

ChromaDB's design prioritizes convenience: calling collection.add(documents=[...]) without specifying embeddings is intended to work, and calling collection.query(query_embeddings=[emb]) without specifying a query text is also intended to work. These two code paths are independent. There is nothing in the API that enforces a relationship between them. A developer upgrading embedding quality — a reasonable engineering decision — naturally starts with the query path (changing how queries are embedded) and plans to re-ingest later. The system continues to return results, so the impact of the incomplete upgrade is not immediately visible.

The Westfield team's investigation focused on query quality improvements — chunking, prompt engineering, n_results tuning — because the system appeared to be functioning. All queries returned results. No exceptions were raised. The recall degradation was only measurable against the ground-truth evaluation set, which had not been run since initial deployment. Establishing a recall@K regression test and running it on every deployment is the only reliable safeguard against this class of silent degradation.

Validation

# Validation log — October 31 2026

# ---- Re-ingestion ----
# 10,000 documents re-ingested with text-embedding-3-small
# Batch size 500 × 20 batches = 10,000 documents
# OpenAI embedding API cost: ~$0.02 (10,000 docs × 50 tokens avg × $0.00002/1k tokens)
# Re-ingestion time: ~45 seconds

# ---- Startup assertion ----
# assert_embedding_consistency() — PASSED
# collection.metadata: {
#   "embedding_model": "text-embedding-3-small",
#   "embedding_dims": 1536,
#   "embedding_provider": "openai",
#   "created_at": "2026-10-31"
# }

# ---- Recall evaluation ----
# Ground-truth set: 100 query/relevant-set pairs
# n_results = 10
#
# Before fix (Sentence Transformers stored + OpenAI query):
#   Recall@10: 21%
#   Mean top-10 cosine similarity: 0.41
#
# After fix (OpenAI stored + OpenAI query):
#   Recall@10: 94%
#   Mean top-10 cosine similarity: 0.74
#   Recovery: +73 percentage points

# ---- Sample query validation ----
# Query: "payment declined on checkout"
# Before fix top-3: ["shipping address update", "password reset", "order cancellation"]
# After fix top-3:  ["card declined at checkout", "payment method rejected", "checkout 422 error"]

# Retainer engagement cost: 1.5h (estimated $112-180 at mid-level rates)

Related retainer posts

Track retainer hours without a client portal

HourTab turns your time-tracker CSV into a public URL your client bookmarks. No login. No portal. CSV in, URL out.

See pricing →