sentence-transformers developer on retainer: indexing pipeline encodes with normalize_embeddings=False — inference service encodes query with normalize_embeddings=True — dot product not cosine similarity — recall@10 drops from 78% to 31% — 3 weeks — sentence-transformers developer on monthly retainer

sentence-transformers 2.7.x • Python 3.11 • all-MiniLM-L6-v2 (384-dim) • 200,000 documents • indexing pipeline: model.encode(texts, normalize_embeddings=False) (default) • document embeddings stored unnormalized • inference service: model.encode([query], normalize_embeddings=True) • query embedding L2-normalized • np.dot(q_normalized, D_unnormalized) computes ||d|| ⋅ cos(θ) not cos(θ) • documents with larger L2 norm score higher regardless of relevance • recall@10 drops 78%→31% • developer checks model version, tokenization, batch size — 3 weeks • fix: re-encode all documents with normalize_embeddings=True • shared encoding config • 1.5h retainer engagement

Eastfield Semantic Search Ltd. built their internal knowledge base search system on sentence-transformers in September 2026, encoding 200,000 technical documents with all-MiniLM-L6-v2. The indexing pipeline was written by developer A, who ran the offline batch encoding job and stored the resulting embeddings as a numpy array on disk. The inference service was written by developer B, who built the real-time query encoding and similarity search endpoint. Developer A used model.encode(texts) with default parameters. Developer B used model.encode([query], normalize_embeddings=True), following documentation examples that recommend normalized embeddings for cosine similarity search. Both services ran without errors, and similarity scores looked reasonable — non-zero, in the range 0.1 to 0.9. But recall@10, measured weekly on a 500-query evaluation set with human-labeled relevant documents, had dropped from 78% at launch to 31% after a routine infrastructure refresh. Developer B spent three weeks checking model version, tokenization parameters, and batch size. They engaged a sentence-transformers retainer developer.

The indexing pipeline and inference service at the time of the engagement:

# indexing.py — Developer A (offline batch job, run once)
# Eastfield Semantic Search Ltd.
# sentence-transformers 2.7.x, Python 3.11

from sentence_transformers import SentenceTransformer
import numpy as np

model = SentenceTransformer("all-MiniLM-L6-v2")

documents = load_all_documents()   # list of 200,000 strings
print(f"Encoding {len(documents)} documents...")

# Default: normalize_embeddings=False
# Document embeddings are raw (not L2-normalized)
doc_embeddings = model.encode(
    documents,
    batch_size=128,
    show_progress_bar=True,
    convert_to_numpy=True,
    # normalize_embeddings=False  ← default; not specified
)
print("doc_embeddings.shape:", doc_embeddings.shape)
# doc_embeddings.shape: (200000, 384)

# Check L2 norms — should be 1.0 if normalized, varies if not
norms = np.linalg.norm(doc_embeddings, axis=1)
print(f"L2 norm: mean={norms.mean():.3f}, min={norms.min():.3f}, max={norms.max():.3f}")
# L2 norm: mean=4.812, min=1.234, max=12.447  ← NOT normalized

np.save("doc_embeddings.npy", doc_embeddings)
print("Saved doc_embeddings.npy")

# ---------------------------------------------------------------------------

# inference.py — Developer B (real-time query endpoint)
# sentence-transformers 2.7.x, Python 3.11

from sentence_transformers import SentenceTransformer
import numpy as np

model = SentenceTransformer("all-MiniLM-L6-v2")
doc_embeddings = np.load("doc_embeddings.npy")   # shape (200000, 384), unnormalized

def search(query: str, top_k: int = 10) -> list[int]:
    """Returns indices of top-k most similar documents."""
    # Developer B uses normalize_embeddings=True (following docs examples)
    query_embedding = model.encode(
        [query],
        normalize_embeddings=True,   # query is L2-normalized: ||q|| = 1.0
        convert_to_numpy=True,
    )[0]  # shape (384,), L2 norm = 1.0

    # Compute dot product between normalized query and stored doc embeddings
    # For normalized q and unnormalized d_i:
    # dot(q, d_i) = ||d_i|| * cos(theta_i)
    # This is NOT cosine similarity — it's cosine similarity scaled by ||d_i||
    # Documents with larger magnitude score higher regardless of angular similarity
    similarities = np.dot(doc_embeddings, query_embedding)   # shape (200000,)
    top_indices = np.argsort(similarities)[::-1][:top_k]
    return top_indices.tolist()

Why normalize_embeddings must match between indexing and inference

Cosine similarity between two vectors q and d is defined as:

cosine_sim(q, d) = dot(q, d) / (||q|| ⋅ ||d||)

When both vectors are L2-normalized (||q|| = 1.0, ||d|| = 1.0), the denominator is 1 and cosine similarity equals the dot product: dot(q_norm, d_norm) = cosine_sim. This is why normalized embeddings with dot product similarity is the standard pattern for efficient semantic search — dot product is fast and BLAS-optimized, and normalization makes it equivalent to cosine similarity.

But when the query is normalized (||q|| = 1.0) and the document embeddings are not normalized (||d_i|| varies), the dot product computes:

dot(q_norm, d_i) = ||d_i|| ⋅ cos(theta_i)

The score is now the product of the cosine similarity angle and the document embedding magnitude. Documents with large L2 norm score higher than documents with small L2 norm, even when the latter are more semantically relevant. In sentence-transformers, embedding magnitude correlates with document length and token frequency — long documents and documents that repeatedly use common tokens tend to produce embeddings with larger L2 norms. This introduces a systematic length bias: long, verbose documents outscore short, precise documents in dot product search, even when the short documents are semantically closer to the query. Recall@10 degrades silently — similarity scores look normal (non-zero, in range), results look plausible (related documents), but the most relevant documents are ranked below longer, tangentially related documents.

# Diagnosing the normalize_embeddings mismatch

from sentence_transformers import SentenceTransformer
import numpy as np

model = SentenceTransformer("all-MiniLM-L6-v2")
doc_embeddings = np.load("doc_embeddings.npy")   # stored embeddings

# Step 1: check stored embedding norms
norms = np.linalg.norm(doc_embeddings, axis=1)
print(f"Stored embedding L2 norms: mean={norms.mean():.3f}")
# Stored embedding L2 norms: mean=4.812
# If norms were ~1.0, embeddings were normalized at indexing time.
# mean=4.812 confirms normalize_embeddings=False was used.

# Step 2: check query embedding norm
query = "how to handle authentication errors"
q_emb = model.encode([query], normalize_embeddings=True, convert_to_numpy=True)[0]
print(f"Query embedding L2 norm: {np.linalg.norm(q_emb):.3f}")
# Query embedding L2 norm: 1.000   ← normalized

# Step 3: demonstrate the scoring bias
# Find a short precise document and a long verbose document
# that both discuss authentication
short_idx = find_short_precise_doc_index("authentication errors")
long_idx = find_long_verbose_doc_index("authentication errors, tokens, headers, ...")

short_norm = np.linalg.norm(doc_embeddings[short_idx])
long_norm  = np.linalg.norm(doc_embeddings[long_idx])

short_cos = np.dot(q_emb, doc_embeddings[short_idx]) / short_norm  # true cosine sim
long_cos  = np.dot(q_emb, doc_embeddings[long_idx])  / long_norm   # true cosine sim

print(f"Short doc: L2 norm={short_norm:.2f}, true cosine_sim={short_cos:.3f}")
print(f"Long doc:  L2 norm={long_norm:.2f},  true cosine_sim={long_cos:.3f}")
# Short doc: L2 norm=2.14, true cosine_sim=0.812
# Long doc:  L2 norm=9.87, true cosine_sim=0.631

short_dot = np.dot(q_emb, doc_embeddings[short_idx])  # 2.14 * 0.812 = 1.74
long_dot  = np.dot(q_emb, doc_embeddings[long_idx])   # 9.87 * 0.631 = 6.23

print(f"Short doc dot product: {short_dot:.3f}")
# Short doc dot product: 1.738  ← higher cosine similarity but lower score
print(f"Long doc dot product:  {long_dot:.3f}")
# Long doc dot product:  6.230  ← lower cosine similarity but higher score
# Long doc ranks higher despite being less semantically relevant

The fix: re-encode all documents with normalize_embeddings=True and use a shared encoding config

Re-encode all 200,000 documents with normalize_embeddings=True. All stored embeddings will then have L2 norm of 1.0, making dot product equivalent to cosine similarity. The inference service already uses normalize_embeddings=True for queries — after re-indexing, both sides are consistent. Add a shared encoding configuration to prevent the mismatch from recurring.

# encoding_config.py — SHARED between indexing.py and inference.py
# Eastfield Semantic Search Ltd.
# sentence-transformers 2.7.x, Python 3.11

ENCODING_CONFIG = {
    "model_name": "all-MiniLM-L6-v2",
    "normalize_embeddings": True,   # MUST match between indexing and inference
    "batch_size": 128,
}
# Store this config alongside the embeddings file so future maintainers
# can verify the encoding settings used to produce the stored index.

import json
with open("doc_embeddings_config.json", "w") as f:
    json.dump(ENCODING_CONFIG, f, indent=2)

# ---------------------------------------------------------------------------

# indexing.py — FIXED (re-encode with normalize_embeddings=True)
from sentence_transformers import SentenceTransformer
from encoding_config import ENCODING_CONFIG
import numpy as np, json

model = SentenceTransformer(ENCODING_CONFIG["model_name"])
documents = load_all_documents()

doc_embeddings = model.encode(
    documents,
    batch_size=ENCODING_CONFIG["batch_size"],
    show_progress_bar=True,
    convert_to_numpy=True,
    normalize_embeddings=ENCODING_CONFIG["normalize_embeddings"],  # True
)
norms = np.linalg.norm(doc_embeddings, axis=1)
assert abs(norms.mean() - 1.0) < 0.001, f"Embeddings not normalized: mean norm={norms.mean()}"

np.save("doc_embeddings.npy", doc_embeddings)
print(f"Saved {len(documents)} normalized embeddings. Mean L2 norm: {norms.mean():.4f}")
# Saved 200000 normalized embeddings. Mean L2 norm: 1.0000

# ---------------------------------------------------------------------------

# inference.py — FIXED (reads shared config, asserts consistency)
from sentence_transformers import SentenceTransformer
from encoding_config import ENCODING_CONFIG
import numpy as np, json

model = SentenceTransformer(ENCODING_CONFIG["model_name"])
doc_embeddings = np.load("doc_embeddings.npy")

# Startup sanity check: verify stored embeddings are normalized
stored_norms = np.linalg.norm(doc_embeddings[:100], axis=1)
if abs(stored_norms.mean() - 1.0) > 0.01:
    raise RuntimeError(
        f"Stored embeddings are NOT normalized (mean L2 norm={stored_norms.mean():.3f}). "
        "Re-run indexing.py with normalize_embeddings=True before starting inference."
    )

def search(query: str, top_k: int = 10) -> list[int]:
    query_embedding = model.encode(
        [query],
        normalize_embeddings=ENCODING_CONFIG["normalize_embeddings"],  # True
        convert_to_numpy=True,
    )[0]
    # Both query and docs are normalized — dot product equals cosine similarity
    similarities = np.dot(doc_embeddings, query_embedding)
    top_indices = np.argsort(similarities)[::-1][:top_k]
    return top_indices.tolist()

Validation

# Validation log — November 2026
# sentence-transformers 2.7.x, Python 3.11, all-MiniLM-L6-v2

# ---- Before fix ----
# Indexing: normalize_embeddings=False (default)
# Stored embedding L2 norms: mean=4.812, min=1.234, max=12.447
# Inference: normalize_embeddings=True for queries
# Similarity metric: np.dot(q_normalized, D_unnormalized)
# Effective metric: ||d_i|| * cos(theta) — length-biased
# Recall@10 on 500-query evaluation set: 31%

# ---- After fix: re-encode with normalize_embeddings=True ----
# Stored embedding L2 norms: mean=1.000, min=0.999, max=1.001  ✓
# Inference: normalize_embeddings=True (unchanged)
# Similarity metric: np.dot(q_normalized, D_normalized) = cosine_sim
# Startup sanity check: mean norm 1.000 — assertion passes             ✓

# ---- Recall evaluation after re-indexing ----
# Recall@10 on 500-query evaluation set: 78%  (returned to baseline)  ✓
# Recall@1 improved from 12% to 51%                                    ✓
# No change in inference latency (same dot product path)                ✓

# ---- Hour breakdown ----
# Triage:     0.5h — recall@10 31% vs 78% baseline; scores look normal;
#             check stored embedding L2 norms: mean 4.812 (not 1.0)
# Root cause: 0.25h — compare normalize_embeddings in indexing.py vs
#             inference.py; indexing False (default) vs inference True;
#             confirm: dot(q_norm, d_unnorm) = ||d|| * cos(theta)
# Fix:        0.5h — re-encode 200,000 docs with normalize_embeddings=True;
#             add shared ENCODING_CONFIG; startup norm assertion
# Validation: 0.25h — recall@10 78%; L2 norms 1.000; startup check passes
# Total:      1.5h

# Retainer engagement cost: 1.5h (estimated $105–165 at mid-level rates)

Related retainer posts

• FAISS developer on retainer — IndexIVFFlat loaded in separate process — make_direct_map() not called after read_index() — index.reconstruct() raises direct_map is not initialized — 3 weeks
• LanceDB developer on retainer — IVF-PQ index built on 50,000 rows — 10,000 new rows not included in index — ANN search blind to new data — 3 weeks
• Voyage AI developer on retainer — query stop-word stripping defeats input_type='query' prefix — recall@10 drops 87%→52% for short queries — 3 weeks