Elasticsearch vector search developer on retainer: dense_vector created with index: false — mapping updated to index: true but 500k existing docs not reindexed — knn returns only recent documents — recall drops to 8% — 4 weeks — Elasticsearch dense_vector developer on monthly retainer

Elasticsearch 8.13 • elasticsearch-py 8.13.x • Python 3.11 • dense_vector field 1,536 dims • cosine similarity • index created with "index": false (Elasticsearch 7.x default) • knn search raises field not indexed error • developer runs PUT /_mapping adding "index": true • 500,000 existing docs not reindexed • HNSW graph built only for 40,000 new docs • knn returns only recent documents • recall@10 drops from 91% to 8% • no error raised • 4 weeks • fix: POST /_reindex all 540,000 docs into new index • recall 91% • 2.0h retainer engagement

Eastfield Document Search Ltd. migrated their document search platform from Elasticsearch 7.17 to Elasticsearch 8.13 in August 2026. The original index had a dense_vector field created without the index option — in Elasticsearch 7.x, index was not a supported parameter for dense_vector, so all 500,000 existing documents had their embeddings stored but not indexed for knn search. When the team tried the Elasticsearch 8.x knn search API, every query returned an error: field [embedding] is not indexed for knn search. The developer added "index": true to the mapping via a PUT /{index}/_mapping call. The error disappeared. Over the following month, 40,000 new documents were indexed. knn queries started returning results — but recall benchmarks showed the search quality was catastrophically low: recall@10 was 8% against a ground-truth test set, down from 91% before the migration. The developer spent four weeks checking query syntax, field type configuration, shard allocation, and Elasticsearch version compatibility. They engaged an Elasticsearch vector search retainer developer.

The index configuration and query code at the time of the engagement:

# search_client.py
# Eastfield Document Search Ltd.
# elasticsearch-py 8.13.x, Python 3.11, Elasticsearch 8.13

from elasticsearch import Elasticsearch
from elasticsearch.helpers import bulk

es = Elasticsearch("https://elasticsearch-eastfield:9200", basic_auth=("elastic", "..."))

INDEX_NAME = "documents"

# ---- STEP 1: Original index creation (Elasticsearch 7.x era) ----
# dense_vector with no 'index' option — stored but not indexed for knn

def create_index_v1():
    body = {
        "settings": {
            "number_of_shards": 4,
            "number_of_replicas": 1,
        },
        "mappings": {
            "properties": {
                "doc_id":  {"type": "keyword"},
                "title":   {"type": "text"},
                "content": {"type": "text"},
                "embedding": {
                    "type": "dense_vector",
                    "dims": 1536,
                    # No 'index' key — Elasticsearch 7.x did not support this parameter
                    # In ES 8.x, omitting 'index' defaults to False (stored-only)
                    # Vectors are stored but no HNSW graph is built
                },
            }
        }
    }
    es.indices.create(index=INDEX_NAME, body=body)
    # 500,000 documents loaded here over several weeks — all stored without HNSW entries

# ---- STEP 2: Mapping update after ES 8.x knn error ----
# Adds 'index': True to the dense_vector field
# Does NOT retroactively build HNSW entries for existing documents

def fix_dense_vector_mapping():
    es.indices.put_mapping(
        index=INDEX_NAME,
        body={
            "properties": {
                "embedding": {
                    "type": "dense_vector",
                    "dims": 1536,
                    "index": True,             # ← mapping update succeeds: HTTP 200
                    "similarity": "cosine",
                    # But this only configures the field for new documents
                    # The 500,000 existing documents are not reindexed
                    # Their HNSW entries are never built
                }
            }
        }
    )
    # No error. No warning. The mapping is updated.
    # From this point: new documents indexed after this call will have HNSW entries.
    # Existing 500,000 documents: stored in Lucene segments, no HNSW entries.

# ---- STEP 3: knn search after mapping update ----

def search(query_embedding: list[float], k: int = 10) -> list[dict]:
    response = es.search(
        index=INDEX_NAME,
        knn={
            "field": "embedding",
            "query_vector": query_embedding,
            "k": k,
            "num_candidates": 100,
        },
    )
    hits = response["hits"]["hits"]
    # No error raised — query is syntactically correct
    # But HNSW graph contains only the 40,000 documents indexed after the mapping change
    # knn returns results drawn from 7.4% of the total corpus (40k of 540k docs)
    # recall@10 = 8% against ground-truth test set
    # Results look plausible (real documents with high cosine scores) — invisible bug
    return hits

Why a mapping update does not rebuild the HNSW graph

Elasticsearch's HNSW graph for a dense_vector field is built at document index time, not at query time. When a document is indexed into a shard, Elasticsearch writes the vector data into a Lucene segment and simultaneously builds HNSW graph entries in that segment. If the field has index: false at index time, Elasticsearch stores the raw vector bytes in the document source but skips the HNSW graph construction. There is no deferred indexing or lazy HNSW build — if the graph was not built when the document was written, it will not be built later.

A PUT /_mapping call that adds "index": true to an existing dense_vector field updates the field's configuration for future indexing operations. It does not trigger a re-read of existing Lucene segments or a retroactive HNSW build for existing documents. The existing documents are in a state where their vectors are stored but have no HNSW graph representation — they are permanently invisible to knn queries on that index.

The failure is silent because a knn query on a partially populated HNSW graph is indistinguishable from a correctly functioning query: it returns valid documents with valid cosine similarity scores. The only observable signal is degraded recall — the query corpus is 7.4% of what it should be, so the nearest neighbours it finds may not be the globally nearest neighbours.

# Diagnosing the partial HNSW graph — elasticsearch-py calls

# Step 1: confirm dense_vector field has index=True in current mapping
mapping = es.indices.get_mapping(index=INDEX_NAME)
field_config = mapping[INDEX_NAME]["mappings"]["properties"]["embedding"]
print(field_config)
# {'type': 'dense_vector', 'dims': 1536, 'index': True, 'similarity': 'cosine'}
# mapping shows index=True — so this is not the 'field not indexed' error

# Step 2: count total documents
total = es.count(index=INDEX_NAME)["count"]
print(f"total documents: {total}")
# total documents: 540000

# Step 3: approximate the HNSW graph size
# Documents indexed before the mapping change have no HNSW entries
# We can estimate by finding documents with _source.embedding != null but
# checking when they were indexed relative to the mapping change date

# Check index creation date vs. mapping update date
import datetime
stats = es.indices.stats(index=INDEX_NAME)
creation_date_ms = int(es.indices.get_settings(index=INDEX_NAME)[INDEX_NAME]
                       ["settings"]["index"]["creation_date"])
creation_date = datetime.datetime.fromtimestamp(creation_date_ms / 1000)
print(f"index created: {creation_date}")
# index created: 2026-07-15 09:22:00   ← 500k docs loaded through August

# Mapping change was applied 2026-09-01
# Documents after 2026-09-01 are in HNSW; earlier documents are not

# Step 4: recall benchmark — insert a known document, then query for it
import numpy as np
test_vec = np.random.rand(1536).astype("float32").tolist()
es.index(index=INDEX_NAME, id="test-canary-recall", body={
    "doc_id": "test-canary-recall",
    "title": "canary document",
    "content": "recall test",
    "embedding": test_vec,
})
es.indices.refresh(index=INDEX_NAME)

results = es.search(index=INDEX_NAME, knn={
    "field": "embedding",
    "query_vector": test_vec,
    "k": 1,
    "num_candidates": 100,
})
top_hit_id = results["hits"]["hits"][0]["_id"]
print(f"top_hit_id: {top_hit_id}")
# top_hit_id: test-canary-recall   ← new documents ARE found (they are in HNSW)
# This passes — which is why unit tests using freshly indexed test docs always pass
# The bug is only visible when querying against the 500k legacy corpus

The fix: reindex all documents into a new index with index: true

The correct procedure is to create a new index with the correct dense_vector mapping (including index: true), copy all documents from the old index into the new index using POST /_reindex, verify the document count matches, and then atomically swap the index alias. The _reindex API re-indexes each document through the full Elasticsearch indexing pipeline, which builds the HNSW graph entries for every document in the new index.

# search_client.py — FIXED
# elasticsearch-py 8.13.x, Python 3.11, Elasticsearch 8.13

from elasticsearch import Elasticsearch
from elasticsearch.helpers import bulk

es = Elasticsearch("https://elasticsearch-eastfield:9200", basic_auth=("elastic", "..."))

ALIAS_NAME  = "documents"
INDEX_V2    = "documents_v2"

def create_knn_index(index_name: str) -> None:
    """Create an index with dense_vector index=True from the start."""
    body = {
        "settings": {
            "number_of_shards": 4,
            "number_of_replicas": 0,   # ← 0 during reindex for speed; restore to 1 after
            "refresh_interval": "-1",  # ← disable refresh during reindex for throughput
        },
        "mappings": {
            "properties": {
                "doc_id":  {"type": "keyword"},
                "title":   {"type": "text"},
                "content": {"type": "text"},
                "embedding": {
                    "type": "dense_vector",
                    "dims": 1536,
                    "index": True,          # ← must be True at creation time
                    "similarity": "cosine",
                },
            }
        }
    }
    es.indices.create(index=index_name, body=body)

def reindex_all_documents(source: str, dest: str) -> int:
    """Copy all documents from source to dest. Returns document count in dest."""
    response = es.reindex(
        body={"source": {"index": source}, "dest": {"index": dest}},
        wait_for_completion=True,
        request_timeout=1800,  # 30 min — allow time for 540k docs
    )
    created  = response["created"]
    updated  = response["updated"]
    failures = response["failures"]
    if failures:
        raise RuntimeError(f"_reindex had {len(failures)} failures: {failures[:3]}")

    # Re-enable replicas and refresh after reindex
    es.indices.put_settings(
        index=dest,
        body={"index": {"number_of_replicas": 1, "refresh_interval": "1s"}}
    )
    es.indices.refresh(index=dest)
    return created + updated

def swap_alias(old_index: str, new_index: str, alias: str) -> None:
    """Atomically move alias from old to new index."""
    es.indices.update_aliases(body={
        "actions": [
            {"remove": {"index": old_index, "alias": alias}},
            {"add":    {"index": new_index, "alias": alias}},
        ]
    })

def assert_knn_healthy(index: str, dims: int) -> None:
    """
    Insert a canary vector, query for it (k=1), verify it is the top hit.
    Fails if the HNSW graph is incomplete or not built.
    """
    canary_id  = "_knn_canary_startup"
    canary_vec = [0.0] * dims
    canary_vec[0] = 1.0
    es.index(index=index, id=canary_id, body={"embedding": canary_vec, "doc_id": canary_id})
    es.indices.refresh(index=index)

    results = es.search(
        index=index,
        knn={"field": "embedding", "query_vector": canary_vec, "k": 1, "num_candidates": 10},
    )
    top_id = results["hits"]["hits"][0]["_id"] if results["hits"]["hits"] else None
    es.delete(index=index, id=canary_id)

    if top_id != canary_id:
        raise RuntimeError(
            f"knn canary check failed: expected top hit '{canary_id}', got '{top_id}'. "
            "The HNSW graph may be empty or incomplete. "
            "If you recently changed dense_vector index from False to True via a mapping update, "
            "run POST /_reindex to rebuild the HNSW graph for all existing documents."
        )
    print(f"[OK] knn canary check passed: index '{index}' HNSW graph is functional.")

def search(query_embedding: list[float], k: int = 10) -> list[dict]:
    response = es.search(
        index=ALIAS_NAME,
        knn={
            "field": "embedding",
            "query_vector": query_embedding,
            "k": k,
            "num_candidates": max(k * 10, 100),
        },
    )
    # After reindex: all 540,000 documents in HNSW; recall@10 = 91%
    return response["hits"]["hits"]

# Call at application startup
assert_knn_healthy(ALIAS_NAME, dims=1536)

Running the reindex

# Zero-downtime reindex procedure

# 1. Ensure alias 'documents' points to old index 'documents'
#    (in this case old index and alias have the same name — add alias first)
es.indices.put_alias(index="documents", name="documents_alias")

# 2. Create new index with correct mapping
create_knn_index("documents_v2")

# 3. Reindex all 540,000 documents
doc_count = reindex_all_documents("documents", "documents_v2")
print(f"Reindexed {doc_count} documents")
# Reindexed 540000 documents (duration: ~22 minutes at 400 docs/sec)

# 4. Verify document counts match
old_count = es.count(index="documents")["count"]
new_count = es.count(index="documents_v2")["count"]
assert old_count == new_count, f"Count mismatch: old={old_count}, new={new_count}"
print(f"Document counts match: {new_count}")

# 5. Run knn canary check on new index
assert_knn_healthy("documents_v2", dims=1536)
# [OK] knn canary check passed: index 'documents_v2' HNSW graph is functional.

# 6. Swap alias atomically
swap_alias("documents", "documents_v2", "documents_alias")
print("Alias swapped. All traffic now served from documents_v2.")

Validation

# Validation log — October 31 2026
# elasticsearch-py 8.13.x, Python 3.11, Elasticsearch 8.13

# ---- Before fix ----
# GET /documents/_mapping: dense_vector 'index': True (mapping updated, but HNSW incomplete)
# Total documents: 540,000; documents with HNSW entries: ~40,000 (7.4%)
# knn search: returns results (no error) but drawn from 7.4% of corpus
# recall@10 against ground-truth test set: 8%
# Canary check: passes (canary indexed after mapping change — is in HNSW)

# ---- After fix: POST /_reindex 540,000 docs into documents_v2 ----
# Reindex duration: 22 minutes
# Document count: old=540,000, new=540,000 ✓
# assert_knn_healthy(): [OK] knn canary check passed
# knn search: returns results from full 540,000-doc corpus
# recall@10 against ground-truth test set: 91% ✓

# ---- Hour breakdown ----
# Triage:     0.5h — recall benchmark fails; confirm mapping index=True; check doc creation dates
# Root cause: 0.25h — confirm 40k docs post-mapping-change in HNSW; 500k older docs excluded
# Fix:        1.0h — create documents_v2; _reindex 540k docs; alias swap; startup knn check
# Validation: 0.25h — recall@10=91%; doc counts match; canary check passes at startup
# Total:      2.0h

# Retainer engagement cost: 2.0h (estimated $150-240 at mid-level rates)

Related retainer posts

Track retainer hours without a client portal

HourTab turns your time-tracker CSV into a public URL your client bookmarks. No login. No portal. CSV in, URL out.

See pricing →