LanceDB developer on retainer: IVF-PQ index built on 50,000 rows — 10,000 new rows added via table.add() not included in existing index — ANN search returns results from indexed fragment only — new data invisible to vector search — 3 weeks — LanceDB developer on monthly retainer

lancedb 0.8.x • Python 3.11 • local LanceDB file-based • 50,000 rows • 1,536 dims • IVF-PQ index built via create_index() • 10,000 new rows added via table.add() afterward • new rows in unindexed fragment • ANN search queries indexed fragment only • 10,000 new rows invisible to vector search • developer investigates query parameters, index config, LanceDB version — 3 weeks • fix: create_index(replace=True) • startup assertion on indexed vs total count • 1.5h retainer engagement

Westfield Local AI Ltd. built their internal document search system on LanceDB in September 2026, choosing it for its zero-infrastructure local deployment and Lance columnar format. Their initial document corpus had 50,000 embedding vectors, and they built an IVF-PQ index with table.create_index() after loading the initial dataset. Queries were fast — 80ms p99 — and returned correct results from the indexed corpus. Two weeks later, the ingestion pipeline added 10,000 new documents via table.add(). table.count_rows() correctly showed 60,000 total rows. But in production, searches for content from the newly added documents consistently returned no matches or irrelevant results. The developer spent three weeks investigating: adjusting query parameters, auditing index configuration, testing different LanceDB versions, and running recall benchmarks. The original 50,000 documents behaved correctly; only the newer 10,000 were invisible. They engaged a LanceDB retainer developer.

The table setup and search pipeline at the time of the engagement:

# search_client.py
# Westfield Local AI Ltd.
# lancedb 0.8.x, Python 3.11, local LanceDB file-based

import lancedb
import pyarrow as pa
import numpy as np

db = lancedb.connect("/data/westfield_lancedb")

# ---- Initial setup: create table and build IVF-PQ index ----

schema = pa.schema([
    pa.field("id", pa.string()),
    pa.field("vector", pa.list_(pa.float32(), 1536)),
    pa.field("text", pa.string()),
    pa.field("source", pa.string()),
])

# Initial load: 50,000 documents
table = db.create_table(
    "documents",
    data=initial_50k_rows,   # list of dicts with id, vector, text, source
    schema=schema,
    mode="create",
)

# Build IVF-PQ index for fast ANN search
table.create_index(
    metric="cosine",
    num_partitions=256,
    num_sub_vectors=96,
)
# Index built over the 50,000 rows present at this point
# table.count_rows() == 50,000
# All 50,000 rows are in the indexed fragment

# ---- Two weeks later: add 10,000 new documents ----

table.add(new_10k_rows)   # 10,000 new dicts with id, vector, text, source
# new_10k_rows written as a NEW fragment (not included in the existing IVF-PQ index)
# table.count_rows() == 60,000  ← rows ARE stored
# But ANN index covers only the 50,000-row indexed fragment

# ---- Search function (broken for new rows) ----

def search(query_embedding: list[float], limit: int = 10) -> list:
    results = (
        table.search(query_embedding)
        .metric("cosine")
        .limit(limit)
        .to_list()
    )
    # Returns results from the 50,000-row indexed fragment
    # The 10,000 new rows in the unindexed fragment are not traversed by IVF-PQ graph
    # New documents never appear in search results
    # No error raised
    return results

Why LanceDB does not auto-update the IVF-PQ index after table.add()

LanceDB's storage model is built on the Lance columnar format, where data is written as immutable fragments. Each call to table.add() writes a new fragment file to disk. When create_index() is called, it builds an IVF-PQ or IVF-FLAT graph over the set of fragments that exist at that moment and writes the index metadata to a separate index directory. The index is a snapshot: it covers exactly the fragments present when create_index() ran.

Fragments added after create_index() are stored in the table and visible to table.count_rows() and to full scans, but they exist outside the index. When an ANN query runs against an indexed table, LanceDB searches the IVF-PQ graph (fast, approximate) over the indexed fragments. It may also perform a sequential scan of unindexed fragments, but for large unindexed portions this scan is either skipped entirely or run with lower recall than the indexed path. The net effect: new documents added after create_index() are invisible to ANN search, even though the rows are physically present in the table.

The failure is silent. table.count_rows() returns the correct count including new rows. table.to_pandas() returns all 60,000 rows. There is no error on table.add() and no warning that the new rows are unindexed. ANN search simply skips the unindexed fragment. The only way to detect the issue is to inspect index statistics and compare indexed row count against total row count.

# Diagnosing the unindexed fragment trap — LanceDB inspection

import lancedb

db = lancedb.connect("/data/westfield_lancedb")
table = db.open_table("documents")

# Step 1: verify total row count
total = table.count_rows()
print(f"Total rows: {total}")
# Total rows: 60000   ← all rows stored, including the 10,000 new ones

# Step 2: inspect index statistics (lancedb 0.8.x)
# The exact API varies by lancedb version — check the release you are running
index_stats = table.index_stats("vector_idx")
print(f"Index stats: {index_stats}")
# Example output:
# {'index_type': 'IVF_PQ', 'metric_type': 'cosine',
#  'num_indexed_rows': 50000,   ← only the original 50,000 rows are indexed
#  'num_unindexed_rows': 10000} ← 10,000 new rows in unindexed fragment

# Step 3: confirm new rows are in the table but not in ANN results
# Full scan finds a known new-batch row
import pyarrow.compute as pc
known_new_id = "doc-50001"
found = table.to_lance().to_table(
    filter=pc.field("id") == known_new_id
)
print(f"Full scan found '{known_new_id}': {len(found)} rows")
# Full scan found 'doc-50001': 1 rows   ← row IS in the table

# ANN search does NOT find it
query_vec = get_embedding_for(known_new_id)  # its own embedding — should be top-1
ann_results = table.search(query_vec).metric("cosine").limit(10).to_list()
found_in_ann = [r for r in ann_results if r["id"] == known_new_id]
print(f"ANN search found '{known_new_id}': {len(found_in_ann)}")
# ANN search found 'doc-50001': 0   ← invisible to ANN search

The fix: rebuild the index with replace=True to cover all fragments

The correct procedure is to call table.create_index(replace=True) to rebuild the IVF-PQ index over all fragments, including the newly added ones. Going forward, a post-add reindex trigger should fire whenever unindexed rows exceed a configured threshold, and a startup assertion should compare table.count_rows() against the indexed row count and fail fast if the gap is too large.

# search_client.py — FIXED
# lancedb 0.8.x, Python 3.11, local LanceDB file-based

import lancedb

db = lancedb.connect("/data/westfield_lancedb")
table = db.open_table("documents")

UNINDEXED_THRESHOLD = 0.10  # rebuild index if >10% of rows are unindexed

def get_indexed_row_count() -> int:
    """Return the number of rows covered by the ANN index."""
    try:
        stats = table.index_stats("vector_idx")
        return stats.get("num_indexed_rows", 0)
    except Exception:
        return 0  # no index present

def assert_index_fresh() -> None:
    """Fail fast if unindexed rows exceed the threshold."""
    total = table.count_rows()
    indexed = get_indexed_row_count()
    unindexed = total - indexed
    fraction = unindexed / total if total > 0 else 0.0

    if fraction > UNINDEXED_THRESHOLD:
        raise RuntimeError(
            f"LanceDB index is stale: {unindexed} of {total} rows ({fraction:.1%}) "
            "are in unindexed fragments and are invisible to ANN search. "
            "Run rebuild_index() to include all rows in the ANN index. "
            "This happens when table.add() is called after create_index() — "
            "LanceDB does not auto-update the IVF-PQ index for new fragments."
        )
    print(f"[OK] Index fresh: {indexed}/{total} rows indexed ({fraction:.1%} unindexed).")

def rebuild_index() -> None:
    """Rebuild the IVF-PQ index over all fragments (including new rows)."""
    total = table.count_rows()
    # num_partitions: rule of thumb is sqrt(n) for IVF
    num_partitions = max(1, int(total ** 0.5))
    table.create_index(
        metric="cosine",
        num_partitions=num_partitions,
        num_sub_vectors=96,
        replace=True,   # ← rebuild over ALL fragments, not just the original ones
    )
    print(f"[OK] Index rebuilt: {total} rows indexed.")

def add_with_reindex(new_rows: list[dict]) -> None:
    """Add rows and rebuild index if unindexed fraction exceeds threshold."""
    table.add(new_rows)
    total = table.count_rows()
    indexed = get_indexed_row_count()
    unindexed = total - indexed
    fraction = unindexed / total if total > 0 else 0.0
    if fraction > UNINDEXED_THRESHOLD:
        print(f"Unindexed rows ({unindexed}/{total}, {fraction:.1%}) exceed threshold — rebuilding index.")
        rebuild_index()

def search(query_embedding: list[float], limit: int = 10) -> list:
    results = (
        table.search(query_embedding)
        .metric("cosine")
        .limit(limit)
        .to_list()
    )
    # After fix: all 60,000 rows indexed; new documents appear in results
    return results

# Call at application startup
assert_index_fresh()

Rebuilding the index for all 60,000 rows

# One-time fix: rebuild index to include 10,000 unindexed rows

import lancedb

db = lancedb.connect("/data/westfield_lancedb")
table = db.open_table("documents")

# Step 1: verify current state
total = table.count_rows()
indexed = get_indexed_row_count()
print(f"Before fix: {indexed} indexed, {total} total, {total - indexed} unindexed")
# Before fix: 50000 indexed, 60000 total, 10000 unindexed

# Step 2: rebuild index over all 60,000 rows
rebuild_index()
# [OK] Index rebuilt: 60000 rows indexed.

# Step 3: verify all rows are now indexed
assert_index_fresh()
# [OK] Index fresh: 60000/60000 rows indexed (0.0% unindexed).

# Step 4: validate that known new-batch document appears in ANN search
known_new_id = "doc-50001"
query_vec = get_embedding_for(known_new_id)
ann_results = table.search(query_vec).metric("cosine").limit(10).to_list()
found = [r for r in ann_results if r["id"] == known_new_id]
print(f"ANN search for '{known_new_id}': {len(found)} result(s)")
# ANN search for 'doc-50001': 1 result(s)   ✓

print("Fix complete. All 60,000 rows visible to ANN search.")

Validation

# Validation log — November 2026
# lancedb 0.8.x, Python 3.11, local LanceDB file-based

# ---- Before fix ----
# table.count_rows(): 60,000  ← all rows stored (no missing rows)
# index_stats: num_indexed_rows=50,000; num_unindexed_rows=10,000
# ANN search for known new-batch IDs: 0 results (invisible to IVF-PQ)
# No errors raised anywhere in the stack

# ---- After fix: create_index(replace=True) over all 60,000 rows ----
# index_stats: num_indexed_rows=60,000; num_unindexed_rows=0             ✓
# assert_index_fresh(): [OK] Index fresh: 60000/60000 rows indexed       ✓
# ANN search for known new-batch doc-50001: appears in top-1             ✓
# ANN search p99 latency: 82ms (same as pre-fix baseline)                ✓
# Recall@10 on held-out test set: 91% (matches expected baseline)        ✓

# ---- Hour breakdown ----
# Triage:     0.5h — recently added documents never appear in ANN results;
#             count_rows()=60,000 confirms rows stored;
#             index_stats: 50,000 indexed vs 60,000 total; 10,000 unindexed
# Root cause: 0.25h — confirm create_index() covers only fragments at build time;
#             new fragments from table.add() not included without replace=True
# Fix:        0.5h — create_index(replace=True) over all 60,000 rows;
#             add_with_reindex() helper; startup assert_index_fresh()
# Validation: 0.25h — index covers 60,000 rows; new-batch docs appear in top-10; 1.5h total

# Retainer engagement cost: 1.5h (estimated $105–165 at mid-level rates)

Related retainer posts

Track retainer hours without a client portal

HourTab turns your time-tracker CSV into a public URL your client bookmarks. No login. No portal. CSV in, URL out.

See pricing →