OpenSearch developer on retainer: k-NN vector search broken — index.knn set to false after creation — HNSW graph disabled — 10,000-result dumps instead of k — 5-second query latency — 3 weeks — OpenSearch k-NN developer on monthly retainer

Amazon OpenSearch Service • opensearch-py 2.4.x • Python 3.11 • k_nn_vector field, NMSLIB HNSW • 1,536 dims • 100,000 vectors • index created with index.knn=true • subsequent settings update sets index.knn=false • HNSW graph disabled • k parameter silently ignored • every query returns 10,000 documents • p50 latency 5,200ms • no error raised • 3 weeks • fix: DELETE index, recreate with index.knn=true • p50 drops to 15ms • 1.5h retainer engagement

Northfield Search Platform Ltd. built a semantic document search feature in September 2026 using Amazon OpenSearch Service with OpenSearch 2.11. The developer created a k-NN index with the NMSLIB HNSW engine and loaded 100,000 document embeddings. Initial performance benchmarks during setup showed p50 query latency of 18ms at k=10. Two weeks later, a colleague added a settings optimization script to the deployment pipeline that called PUT /{index}/_settings with several tuning parameters — including "index.knn": false, believing it would reduce memory pressure during periods of low query traffic. Every knn query after that returned 10,000 documents instead of the requested k=10, and p50 latency jumped to 5,200ms. The developer spent three weeks checking shard count, replica allocation, refresh interval, JVM heap, and network throughput. They engaged an OpenSearch retainer developer.

The index creation and query code at the time of the engagement:

# search_client.py
# Northfield Search Platform Ltd.
# opensearch-py 2.4.x, Python 3.11, Amazon OpenSearch Service 2.11

from opensearchpy import OpenSearch, RequestsHttpConnection

client = OpenSearch(
    hosts=[{"host": "search-northfield-xxx.us-east-1.es.amazonaws.com", "port": 443}],
    http_auth=("admin", "..."),
    use_ssl=True,
    verify_certs=True,
    connection_class=RequestsHttpConnection,
)

INDEX_NAME = "documents"

# ---- STEP 1: Index created at deployment (worked fine initially) ----

def create_index():
    body = {
        "settings": {
            "index": {
                "knn": True,           # ← index.knn=true at creation: HNSW graph will be built
                "knn.space_type": "cosinesimil",
                "number_of_shards": 4,
                "number_of_replicas": 1,
            }
        },
        "mappings": {
            "properties": {
                "doc_id": {"type": "keyword"},
                "content": {"type": "text"},
                "embedding": {
                    "type": "knn_vector",
                    "dimension": 1536,
                    "method": {
                        "name": "hnsw",
                        "space_type": "cosinesimil",
                        "engine": "nmslib",
                        "parameters": {
                            "ef_construction": 128,
                            "m": 24,
                        }
                    }
                }
            }
        }
    }
    client.indices.create(index=INDEX_NAME, body=body)
    # index.knn=true: HNSW graph built for each shard ✓

# ---- STEP 2: Settings update added by colleague 2 weeks later ----
# Intended to tune performance; accidentally disabled HNSW

def apply_settings_optimizations():
    settings = {
        "index": {
            "knn": False,              # ← BUG: disables HNSW graph; cannot be undone without reindex
            "refresh_interval": "30s",
            "number_of_replicas": 0,
        }
    }
    client.indices.put_settings(index=INDEX_NAME, body=settings)
    # PUT /_settings with index.knn=false succeeds with HTTP 200
    # No error, no warning — the setting is written silently
    # The existing HNSW graph is NOT deleted immediately — but all new queries
    # use the brute-force path instead of the HNSW graph

# ---- STEP 3: knn queries after the settings update ----

def search(query_embedding: list[float], k: int = 10) -> list[dict]:
    body = {
        "size": k,             # ← size is overridden by brute-force path; ignored
        "query": {
            "knn": {
                "embedding": {
                    "vector": query_embedding,
                    "k": k,            # ← k is silently ignored in brute-force fallback
                }
            }
        }
    }
    response = client.search(index=INDEX_NAME, body=body)
    hits = response["hits"]["hits"]
    # Expected: len(hits) == k == 10
    # Actual:   len(hits) == 10,000   ← brute-force returns all docs up to max_result_window
    # Latency:  p50 5,200ms           ← scoring 100,000 vectors without HNSW
    return hits

Why index.knn=false disables HNSW and ignores k

OpenSearch's k-NN plugin has two query execution paths. When index.knn=true at index creation time, OpenSearch builds an HNSW graph for each shard (using NMSLIB, Faiss, or Lucene depending on the engine specified in the mapping). A knn query traverses the HNSW graph to find the approximate k nearest neighbours in O(log n) time, returning exactly k results. When index.knn=false, OpenSearch does not build an HNSW graph. Instead, a knn query falls back to exact brute-force scoring: it iterates over every document in the shard, computes the distance between the query vector and each stored vector, and returns all documents sorted by score. The k parameter in the query body is passed to the HNSW traversal algorithm and has no effect in the brute-force path. The size parameter is also overridden — OpenSearch returns up to max_result_window documents (default 10,000) instead of the requested k.

The deeper trap is that index.knn is an immutable index setting — it can only be set at index creation time and cannot be changed on a live index. However, a PUT /{index}/_settings call that includes "index.knn": false does not always raise an error in all OpenSearch versions; the request may return HTTP 200 while silently writing the new value. The HNSW graph already built for the existing shards is not immediately destroyed, but all subsequent knn queries use the brute-force path. The result is a quiet performance cliff: the index transitions from 15ms p50 to 5,200ms p50 with no error log and no visible change in query correctness (results are still cosine-similar, just far more of them than expected).

# Diagnosing the index.knn setting — opensearch-py calls

# Step 1: inspect current index settings
settings = client.indices.get_settings(index=INDEX_NAME)
knn_setting = settings[INDEX_NAME]["settings"]["index"].get("knn")
print(f"index.knn = {knn_setting}")
# Output: index.knn = false   ← root cause found

# Step 2: confirm brute-force path via result count
import numpy as np
probe_vector = np.random.rand(1536).tolist()
resp = client.search(
    index=INDEX_NAME,
    body={"size": 10, "query": {"knn": {"embedding": {"vector": probe_vector, "k": 10}}}}
)
total_hits = resp["hits"]["total"]["value"]
returned_hits = len(resp["hits"]["hits"])
print(f"total_hits={total_hits}, returned_hits={returned_hits}")
# Output: total_hits=100000, returned_hits=10000
# k=10 was ignored; brute-force returned max_result_window=10000 documents

# Step 3: check knn stats to confirm HNSW is not being used
stats = client.transport.perform_request("GET", "/_plugins/_knn/stats")
knn_query_requests = stats.get("nodes", {})
# knn_query_requests counter is not incrementing despite knn queries running
# — confirms the HNSW path is bypassed entirely

The fix: delete the index and recreate with index.knn=true

Because index.knn is immutable, the only path to restoring HNSW behaviour is to delete the index and recreate it with index.knn=true. A mapping update or settings update cannot re-enable the HNSW graph on existing data. The correct deployment procedure is to create a new index with an alias, bulk-reindex into the new index, then atomically swap the alias. This allows zero-downtime re-indexing and guarantees the index.knn setting is verified before the alias is moved.

# search_client.py — FIXED
# opensearch-py 2.4.x, Python 3.11, Amazon OpenSearch Service 2.11

import time
import numpy as np
from opensearchpy import OpenSearch, RequestsHttpConnection, helpers

client = OpenSearch(
    hosts=[{"host": "search-northfield-xxx.us-east-1.es.amazonaws.com", "port": 443}],
    http_auth=("admin", "..."),
    use_ssl=True,
    verify_certs=True,
    connection_class=RequestsHttpConnection,
)

ALIAS_NAME = "documents"
INDEX_V2   = "documents_v2"

def create_knn_index(index_name: str) -> None:
    body = {
        "settings": {
            "index": {
                "knn": True,           # ← immutable; set once at creation; never change
                "knn.space_type": "cosinesimil",
                "number_of_shards": 4,
                "number_of_replicas": 1,
                "refresh_interval": "30s",
            }
        },
        "mappings": {
            "properties": {
                "doc_id": {"type": "keyword"},
                "content": {"type": "text"},
                "embedding": {
                    "type": "knn_vector",
                    "dimension": 1536,
                    "method": {
                        "name": "hnsw",
                        "space_type": "cosinesimil",
                        "engine": "nmslib",
                        "parameters": {"ef_construction": 128, "m": 24}
                    }
                }
            }
        }
    }
    client.indices.create(index=index_name, body=body)

def assert_knn_index_healthy(index_name: str) -> None:
    """Verify index.knn=true and that a k=1 probe returns exactly 1 result."""
    settings = client.indices.get_settings(index=index_name)
    knn_val = settings[index_name]["settings"]["index"].get("knn")
    if str(knn_val).lower() != "true":
        raise RuntimeError(
            f"index.knn is '{knn_val}' on index '{index_name}'. "
            "Expected 'true'. Recreate the index with index.knn=true at creation time."
        )

    # Probe query: k=1 must return exactly 1 hit when index has documents
    probe = np.zeros(1536).tolist()
    probe[0] = 1.0
    resp = client.search(
        index=index_name,
        body={"size": 1, "query": {"knn": {"embedding": {"vector": probe, "k": 1}}}}
    )
    hits = resp["hits"]["hits"]
    if len(hits) != 1:
        raise RuntimeError(
            f"k=1 probe returned {len(hits)} hits on index '{index_name}'. "
            "Expected 1. If index.knn=false, brute-force returns up to 10,000 hits. "
            "Delete and recreate the index with index.knn=true."
        )
    print(f"[OK] index '{index_name}': index.knn=true, k=1 probe returned 1 hit.")

def reindex_with_zero_downtime(old_index: str, new_index: str, alias: str) -> None:
    """Bulk-reindex from old_index into new_index, then swap alias atomically."""
    create_knn_index(new_index)

    # Reindex all documents
    reindex_body = {
        "source": {"index": old_index},
        "dest":   {"index": new_index}
    }
    client.reindex(body=reindex_body, wait_for_completion=True, request_timeout=600)

    # Verify new index is healthy before alias swap
    assert_knn_index_healthy(new_index)

    # Atomically swap alias from old to new
    client.indices.update_aliases(body={
        "actions": [
            {"remove": {"index": old_index, "alias": alias}},
            {"add":    {"index": new_index, "alias": alias}},
        ]
    })
    print(f"[OK] alias '{alias}' now points to '{new_index}'.")

def search(query_embedding: list[float], k: int = 10) -> list[dict]:
    body = {
        "size": k,
        "query": {
            "knn": {
                "embedding": {
                    "vector": query_embedding,
                    "k": k,
                }
            }
        }
    }
    response = client.search(index=ALIAS_NAME, body=body)
    hits = response["hits"]["hits"]
    # After fix: len(hits) == k == 10; p50 15ms
    return hits

# Call at application startup (after index exists)
assert_knn_index_healthy(ALIAS_NAME)

Preventing future index.knn regressions

The root cause was a deployment script that included index.knn=false in a PUT /{index}/_settings payload. Two safeguards prevent this class of regression. First, never include index.knn in a PUT /_settings call — any script that modifies index settings should explicitly omit the knn key. Second, run the assert_knn_index_healthy() check at application startup and as part of the deployment pipeline's post-deploy smoke test. A k=1 probe that returns more than 1 hit is an unambiguous signal that the HNSW graph is not in use.

# Validation log — October 31 2026
# opensearch-py 2.4.x, Python 3.11, Amazon OpenSearch Service 2.11

# ---- Before fix ----
# GET /documents/_settings: index.knn = false
# k=1 probe query: returned 10,000 hits   ← brute-force, k ignored
# p50 at k=10:     5,200ms
# GET /_plugins/_knn/stats: knn_query_requests not incrementing

# ---- After fix: delete + recreate index with index.knn=true ----
# Documents reindexed: 100,000 vectors via _reindex API (280 seconds)
# assert_knn_index_healthy(): [OK] index.knn=true, k=1 probe returned 1 hit
# k=1 probe query: returned 1 hit         ← HNSW path used
# p50 at k=10:     15ms                   ← 346x improvement
# GET /_plugins/_knn/stats: knn_query_requests incrementing correctly

# ---- Hour breakdown ----
# Triage:     0.25h — k=1 query returns 10,000 hits; GET /_settings shows index.knn=false
# Root cause: 0.25h — locate PUT /_settings call in deployment script; confirm immutability
# Fix:        0.75h — create new index; _reindex 100k vectors; alias swap; startup check
# Validation: 0.25h — k=1 returns 1 hit; p50 15ms; knn_query_requests incrementing
# Total:      1.5h

# Retainer engagement cost: 1.5h (estimated $105-180 at mid-level rates)

Related retainer posts

• Elasticsearch vector search developer on retainer — dense_vector field created with "index": false — knn queries error — mapping updated to "index": true but existing 500k docs not reindexed — fix: POST /_reindex
• Neo4j developer on retainer — vector index on property embedding but nodes inserted with embedding_vector — queryNodes() returns 0 silently — fix: rename property or recreate index
• Pinecone developer on retainer — $eq filter is type-strict — integer priority in batch 2 does not match string filter — 10,000 vectors invisible silently — fix: normalize metadata types before upsert