Redis vector search developer on retainer: HNSW index on HASH type but vectors inserted as JSON — RediSearch silently ignores JSON fields — 0 results on every query — 2 weeks — Redis Stack RediSearch vector search developer on monthly retainer

Redis Stack (RediSearch module) • redis-py 5.0.x • Python 3.11 • FLAT index created ON HASH • 60,000 documents • 1536 dimensions • cosine distance • documents inserted via client.json().set() (JSON type) instead of client.hset() (HASH type) • RediSearch ON HASH index silently skips JSON-type keys • docs_indexed=0 after 60,000 inserts • every FT.SEARCH KNN query returns 0 results • no error raised • 2 weeks • fix: re-ingest via hset() with embedding serialized as float32 bytes • 1.5h retainer engagement

Eastfield Cache Layer Ltd. added a semantic document search feature in October 2026 using Redis Stack's vector similarity search. The ingestion pipeline inserted 60,000 documents using the Redis JSON module. The FT.SEARCH KNN query returned zero results for every query vector. The developer verified the HNSW parameters, the cosine distance metric, the vector dimensions, and the KNN query syntax. Every query returned empty results. They engaged a Redis retainer developer.

The RediSearch index definition and the document insertion code at the time of the engagement:

# vector_store.py
# Eastfield Cache Layer Ltd.
# redis-py 5.0.x, Python 3.11, Redis Stack (RediSearch + RedisJSON modules)

import redis
import numpy as np

client = redis.Redis(host="localhost", port=6379, decode_responses=True)
# decode_responses=True — all responses decoded as UTF-8 strings
# BUG: this setting also decodes binary HASH fields — never use with hset() for vectors

# Index creation — run once at startup:
# FT.CREATE idx ON HASH PREFIX 1 doc:
#   SCHEMA
#     title TEXT
#     content TEXT
#     embedding VECTOR FLAT 6
#       TYPE FLOAT32
#       DIM 1536
#       DISTANCE_METRIC COSINE
#
# Key point: ON HASH means the index ONLY scans HASH-type keys.
# JSON-type keys with the same prefix are completely invisible to this index.

def create_index(client: redis.Redis) -> None:
    try:
        client.execute_command(
            "FT.CREATE", "idx",
            "ON", "HASH",            # ← index only reads HASH-type keys
            "PREFIX", "1", "doc:",
            "SCHEMA",
            "title", "TEXT",
            "content", "TEXT",
            "embedding", "VECTOR", "FLAT", "6",
            "TYPE", "FLOAT32",
            "DIM", "1536",
            "DISTANCE_METRIC", "COSINE",
        )
        print("Index 'idx' created ON HASH")
    except redis.exceptions.ResponseError as e:
        if "Index already exists" in str(e):
            print("Index 'idx' already exists")
        else:
            raise

def insert_documents(documents: list[dict]) -> int:
    # BUG: using the JSON module to store documents
    # client.json().set() stores keys as JSON type
    # The FT.CREATE index ON HASH does not scan JSON-type keys
    # Documents are inserted successfully — Redis stores them as JSON
    # But the RediSearch vector index never reads them
    for doc in documents:
        client.json().set(
            f"doc:{doc['id']}",    # key matches the index prefix "doc:"
            "$",                    # JSON root path
            {
                "title": doc["title"],
                "content": doc["content"],
                "embedding": doc["embedding"],   # list[float]
            }
        )
        # TYPE doc:1 → ReJSON-RL (JSON type)   ← wrong type for ON HASH index
        # FT.INFO idx → docs_indexed: 0          ← index sees 0 matching keys
    return len(documents)   # returns 60,000 — JSON keys stored, but not indexed

def search_documents(query_embedding: list[float], top_k: int = 10) -> list[dict]:
    # Serialize query vector for FT.SEARCH KNN
    query_bytes = np.array(query_embedding, dtype=np.float32).tobytes()

    # FT.SEARCH KNN query — correct syntax
    results = client.execute_command(
        "FT.SEARCH", "idx",
        f"*=>[KNN {top_k} @embedding $vec AS score]",
        "SORTBY", "score", "ASC",
        "RETURN", "3", "title", "content", "score",
        "PARAMS", "2", "vec", query_bytes,
        "DIALECT", "2",
    )
    # results == [0]   ← total_results: 0, no documents returned
    # The KNN query is syntactically correct
    # The index has 0 indexed documents because all keys are JSON type, not HASH type
    return []

Why RediSearch vector indexes only work on HASH type (or JSON type — not both)

Redis Stack supports two document types: HASH (the traditional Redis data structure) and JSON (the RedisJSON module type). A RediSearch index is created with an explicit ON HASH or ON JSON clause that binds the index to one document type. The index scanner watches for key-write events matching the configured prefix and type. If a key with the right prefix is stored as the wrong type, the scanner silently skips it — there is no error, no warning, and no counter that increments to indicate skipped documents.

The FT.INFO command reports docs_indexed: 0 when all documents have been skipped. This is easy to miss in verbose output. The developer looked at the index name, the field schema, and the HNSW parameters in FT.INFO — but not at the docs_indexed counter three lines below. Once the counter is verified, the root cause is immediately clear: the index is empty because the documents are the wrong type.

This bug has a compounding factor: decode_responses=True in the redis-py client. This setting decodes all byte responses from Redis as UTF-8 strings, which is convenient for text fields but breaks vector storage. HASH fields storing float32 bytes cannot be round-tripped through UTF-8 decoding. The correct approach is to use decode_responses=False (or use a separate client with decode_responses=False specifically for vector operations).

# Diagnosing the HASH vs JSON mismatch
# redis-py 5.0.x, Python 3.11, Redis Stack

import redis

client = redis.Redis(host="localhost", port=6379, decode_responses=True)

# Step 1: Check FT.INFO for docs_indexed
info = client.execute_command("FT.INFO", "idx")
info_dict = dict(zip(info[::2], info[1::2]))
print(f"docs_indexed: {info_dict.get('docs_indexed', 'N/A')}")
# Output: docs_indexed: 0   ← root cause immediately visible

print(f"max_doc_id: {info_dict.get('max_doc_id', 'N/A')}")
# Output: max_doc_id: 0   ← confirms: no documents were ever indexed

# Step 2: Verify the type of stored keys
key_type = client.type("doc:1")
print(f"TYPE doc:1 → {key_type}")
# Output: TYPE doc:1 → ReJSON-RL   ← JSON type, not HASH type
# The FT.CREATE index ON HASH never scans ReJSON-RL keys

# Step 3: Verify what data is in the key (JSON-style)
json_client = redis.Redis(host="localhost", port=6379)
json_module = json_client.json()
doc = json_module.get("doc:1", "$")
print(f"doc:1 (JSON): {doc}")
# Output: [{'title': 'Example Document', 'content': '...', 'embedding': [0.1, 0.2, ...]}]
# The data IS in Redis — stored as JSON — but the HASH index ignores it

# Step 4: Confirm a HASH key would be indexed
import numpy as np
hash_client = redis.Redis(host="localhost", port=6379, decode_responses=False)
# decode_responses=False required for binary HASH fields
test_embedding = np.array([0.1] * 1536, dtype=np.float32).tobytes()
hash_client.hset("doc:test_hash", mapping={
    "title": "Test Hash Document",
    "content": "This is a HASH-type key",
    "embedding": test_embedding,
})
# TYPE doc:test_hash → hash   ← correct type for ON HASH index

# Check FT.INFO again after inserting one HASH key
info = client.execute_command("FT.INFO", "idx")
info_dict = dict(zip(info[::2], info[1::2]))
print(f"docs_indexed after HASH insert: {info_dict.get('docs_indexed', 'N/A')}")
# Output: docs_indexed after HASH insert: 1   ← confirmed: HASH keys are indexed

The fix: re-ingest via hset() with embedding serialized as float32 bytes

# vector_store.py — FIXED
# redis-py 5.0.x, Python 3.11, Redis Stack (RediSearch module)

import redis
import numpy as np

# Use decode_responses=False for the client that handles vector operations
# decode_responses=True would corrupt the binary float32 bytes in HASH fields
vector_client = redis.Redis(host="localhost", port=6379, decode_responses=False)

# Text-only client for reading string fields (title, content)
text_client = redis.Redis(host="localhost", port=6379, decode_responses=True)

def insert_documents_fixed(documents: list[dict]) -> int:
    """
    Inserts documents as HASH-type keys using hset().
    The FT.CREATE ON HASH index will scan and index these keys.
    """
    pipe = vector_client.pipeline(transaction=False)
    for doc in documents:
        key = f"doc:{doc['id']}"
        # Serialize the float list to bytes: must match the index's TYPE FLOAT32
        embedding_bytes = np.array(doc["embedding"], dtype=np.float32).tobytes()
        pipe.hset(key, mapping={
            b"title": doc["title"].encode("utf-8"),
            b"content": doc["content"].encode("utf-8"),
            b"embedding": embedding_bytes,   # binary float32 array
        })
    pipe.execute()
    # TYPE doc:1 → hash   ← correct type; ON HASH index will scan and index this key
    return len(documents)

def assert_docs_indexed(expected_count: int, index_name: str = "idx", tolerance: float = 0.01) -> None:
    """
    Verifies that RediSearch has indexed the expected number of documents.
    Fails if docs_indexed is 0 or significantly below expected_count.
    """
    info = text_client.execute_command("FT.INFO", index_name)
    info_dict = dict(zip(info[::2], info[1::2]))
    docs_indexed = int(info_dict.get("docs_indexed", 0))
    indexing_failures = int(info_dict.get("indexing_failures", 0))

    if docs_indexed == 0 and expected_count > 0:
        on_clause = "unknown"
        for i, val in enumerate(info):
            if val == "on":
                on_clause = info[i + 1] if i + 1 < len(info) else "unknown"
                break
        raise RuntimeError(
            f"RediSearch index '{index_name}' has 0 docs_indexed after inserting "
            f"{expected_count} documents. Most likely cause: FT.CREATE ON {on_clause.upper()} "
            f"but documents were inserted with the wrong Redis data type. "
            f"Check: TYPE  — should be 'hash' for ON HASH indexes. "
            f"Fix: re-ingest all documents using hset() with embedding as float32 bytes."
        )

    min_expected = int(expected_count * (1 - tolerance))
    if docs_indexed < min_expected:
        raise RuntimeError(
            f"RediSearch index '{index_name}' has only {docs_indexed} docs_indexed "
            f"(expected {expected_count}, tolerance {tolerance*100:.0f}%). "
            f"Indexing failures: {indexing_failures}. "
            f"Check FT.INFO for error details."
        )

    print(f"[OK] Index '{index_name}': {docs_indexed} docs indexed (expected {expected_count}).")

def search_documents_fixed(query_embedding: list[float], top_k: int = 10) -> list[dict]:
    query_bytes = np.array(query_embedding, dtype=np.float32).tobytes()
    results = vector_client.execute_command(
        "FT.SEARCH", "idx",
        f"*=>[KNN {top_k} @embedding $vec AS score]",
        "SORTBY", "score", "ASC",
        "RETURN", "3", "title", "content", "score",
        "PARAMS", "2", "vec", query_bytes,
        "DIALECT", "2",
    )
    total = results[0]
    if total == 0:
        return []

    docs = []
    for i in range(1, len(results), 2):
        key = results[i]
        fields = dict(zip(results[i+1][::2], results[i+1][1::2]))
        docs.append({
            "id": key.decode("utf-8").replace("doc:", ""),
            "title": fields.get(b"title", b"").decode("utf-8"),
            "content": fields.get(b"content", b"").decode("utf-8"),
            "score": float(fields.get(b"score", b"0")),
        })
    return docs

# Re-ingestion script — run once to migrate all existing JSON keys to HASH keys
# redis-py 5.0.x, Python 3.11

import redis
import numpy as np

text_client = redis.Redis(host="localhost", port=6379, decode_responses=True)
vector_client = redis.Redis(host="localhost", port=6379, decode_responses=False)

def migrate_json_to_hash(prefix: str = "doc:") -> int:
    """
    Finds all keys with the given prefix that are of type JSON (ReJSON-RL),
    reads their data, deletes them, and re-inserts as HASH type.
    Returns the number of migrated documents.
    """
    migrated = 0
    cursor = 0
    json_client = text_client.json()
    pipe = vector_client.pipeline(transaction=False)
    batch_size = 500

    while True:
        cursor, keys = text_client.scan(cursor, match=f"{prefix}*", count=500)
        for key in keys:
            key_type = text_client.type(key)
            if key_type == "ReJSON-RL":
                # Read the JSON document
                doc = json_client.get(key, "$")
                if doc and isinstance(doc, list) and len(doc) > 0:
                    doc = doc[0]
                    embedding = doc.get("embedding", [])
                    embedding_bytes = np.array(embedding, dtype=np.float32).tobytes()
                    # Delete JSON key
                    text_client.delete(key)
                    # Re-insert as HASH
                    pipe.hset(key.encode(), mapping={
                        b"title": doc.get("title", "").encode("utf-8"),
                        b"content": doc.get("content", "").encode("utf-8"),
                        b"embedding": embedding_bytes,
                    })
                    migrated += 1
                    if migrated % batch_size == 0:
                        pipe.execute()
                        pipe = vector_client.pipeline(transaction=False)

        if cursor == 0:
            break

    if migrated % batch_size != 0:
        pipe.execute()   # flush remaining batch

    return migrated

# Run migration
count = migrate_json_to_hash("doc:")
print(f"Migrated {count} documents from JSON to HASH type")

# Verify docs_indexed after migration
assert_docs_indexed(expected_count=count, index_name="idx")

Alternative: use ON JSON index instead of ON HASH

If the application's original intent was to store documents as JSON (for easier partial updates and nested field access), the correct solution is to recreate the RediSearch index with ON JSON instead of ON HASH — and to use the JSON path syntax for the vector field in the schema definition. This keeps the existing JSON keys and avoids the re-ingestion step.

# Alternative fix: recreate the index ON JSON (no re-ingestion needed)
# Drop the existing index:
# FT.DROPINDEX idx

# Recreate ON JSON with the embedding field path using JSON dot notation:
client.execute_command(
    "FT.CREATE", "idx_json",
    "ON", "JSON",                     # ← now indexes JSON-type keys
    "PREFIX", "1", "doc:",
    "SCHEMA",
    "$.title", "AS", "title", "TEXT",
    "$.content", "AS", "content", "TEXT",
    "$.embedding", "AS", "embedding", "VECTOR", "FLAT", "6",
    "TYPE", "FLOAT32",
    "DIM", "1536",
    "DISTANCE_METRIC", "COSINE",
)

# No re-ingestion needed: all existing JSON keys are now indexed by idx_json
# FT.INFO idx_json → docs_indexed: 60,000   ← immediately visible after index builds

# FT.SEARCH syntax is the same; the JSON path is handled internally:
def search_documents_json_index(query_embedding: list[float], top_k: int = 10) -> list:
    query_bytes = np.array(query_embedding, dtype=np.float32).tobytes()
    results = vector_client.execute_command(
        "FT.SEARCH", "idx_json",
        f"*=>[KNN {top_k} @embedding $vec AS score]",
        "SORTBY", "score", "ASC",
        "RETURN", "3", "title", "content", "score",
        "PARAMS", "2", "vec", query_bytes,
        "DIALECT", "2",
    )
    return results

# Trade-off: ON JSON supports nested documents and partial updates via json().set()
# ON HASH is simpler for flat documents and slightly faster for read-heavy workloads
# Choose based on your update pattern — pick one and stay consistent

Validation

# Validation log — October 31 2026
# redis-py 5.0.x, Python 3.11, Redis Stack (RediSearch module)

# ---- Before fix ----
# FT.INFO idx: docs_indexed=0, max_doc_id=0   ← index empty
# TYPE doc:1 → ReJSON-RL   ← JSON type; ON HASH index ignores it
# FT.SEARCH KNN query: total_results=0   ← 0 results for every query vector
# Developer verified numCandidates equivalent (KNN K value), DISTANCE_METRIC, DIM — all correct

# ---- Diagnosis ----
# FT.INFO: docs_indexed=0   ← visible immediately; root cause confirmed
# TYPE doc:1: ReJSON-RL   ← JSON type vs ON HASH index = mismatch
# After inserting one HASH test key: docs_indexed=1   ← confirmed HASH keys are indexed

# ---- After fix: migrate_json_to_hash() re-ingestion ----
# Migrated 60,000 documents from JSON to HASH type
# FT.INFO idx: docs_indexed=60,000   ← all documents indexed
# assert_docs_indexed(60,000): [OK] Index 'idx': 60,000 docs indexed (expected 60,000).
# FT.SEARCH KNN query with query_embedding=[0.5]*1536:
#   total_results=60,000; top 10 returned with score < 0.02   ← correct high-similarity results
# All queries return relevant results; no empty result sets

# ---- Alternative fix: ON JSON index ----
# FT.CREATE idx_json ON JSON: index built in ~3 minutes for 60,000 documents
# FT.INFO idx_json: docs_indexed=60,000
# search_documents_json_index(): returns 10 results   ← confirmed working

# ---- Hour breakdown ----
# Triage: 0.5h — FT.INFO; confirm docs_indexed=0; check TYPE of stored keys; confirm ReJSON-RL vs HASH
# Root cause: 0.25h — insert one HASH test key; confirm docs_indexed increments; verify mismatch
# Fix: 0.5h — write and run migrate_json_to_hash(); update insert_documents() to use hset(); add assertion
# Validation: 0.25h — docs_indexed=60,000; KNN queries return correct results; smoke test passes
# Total: 1.5h

# Retainer engagement cost: 1.5h (estimated $105-165 at mid-level rates)

Related retainer posts

• MongoDB developer on retainer — Atlas Vector Search index on field embedding but documents use field embeddings — 0 results silently — fix: recreate index on correct field
• Supabase developer on retainer — RLS enabled on documents table with no policy — every INSERT silently ignored — fix: CREATE POLICY
• Milvus developer on retainer — collection.search() raises MilvusException: collection not loaded after create_index() — collection.load() missing