Turbopuffer developer on retainer: schema inference locks timestamp as text type — integer timestamps in second batch coerced to string silently — numeric $gte/$lte filter operators fail for all second-batch documents — 3 weeks — Turbopuffer developer on monthly retainer

Turbopuffer Cloud serverless • turbopuffer Python SDK latest • Python 3.11 • 100,000 vectors • namespace with schema inference • first batch stores timestamp as ISO 8601 string • schema locked as text • second batch stores timestamp as Unix integer • integer coerced to string silently • numeric $gte/$lte filters fail for second-batch documents • developer investigates query vectors, similarity metric, filter syntax — 3 weeks • fix: delete namespace • re-ingest with consistent integer timestamps • 1.5h retainer engagement

Northfield Fast Search Ltd. built their event search system on Turbopuffer in August 2026, choosing it for its serverless model and fast filtered vector search. Their event corpus had 100,000 documents ingested in two batches from separate services: a historical backfill from the data warehouse (50,000 documents) and a real-time ingestion pipeline from their event processing service. The historical backfill stored event timestamps as ISO 8601 strings: "2026-09-15T12:00:00Z". The real-time pipeline, written by a different team member, stored timestamps as Unix integers: 1726401600. Both batches completed without error. But in production, date-range filtered searches that used $gte and $lte operators on the timestamp attribute returned zero results for all events from the real-time pipeline. Only historical events matched date-range filters — and even those only matched when queried without numeric operators. The developer spent three weeks adjusting filter syntax, testing query vectors, and auditing similarity metric configuration. They engaged a Turbopuffer retainer developer.

The namespace setup and upsert pipeline at the time of the engagement:

# search_client.py
# Northfield Fast Search Ltd.
# turbopuffer Python SDK latest, Python 3.11, Turbopuffer Cloud

import turbopuffer as tpuf

ns = tpuf.Namespace("fast_search_events")

# ---- Historical backfill (50,000 documents — data warehouse service) ----

def upsert_historical_batch(docs: list[dict]) -> None:
    """
    docs: list of {"id": str, "vector": list[float],
                   "timestamp": str, "category": str}
    timestamp stored as ISO 8601 string: "2026-09-15T12:00:00Z"
    """
    ns.upsert(
        ids=[d["id"] for d in docs],
        vectors=[d["vector"] for d in docs],
        attributes={
            "timestamp": [d["timestamp"] for d in docs],
            # e.g. "2026-09-15T12:00:00Z"
            # Turbopuffer schema-infers timestamp as type: text
            "category": [d["category"] for d in docs],
        },
        distance_metric="cosine_distance",
    )
    # Upsert succeeds; no error raised
    # ns.schema() after this call:
    # {"timestamp": "text", "category": "text"}

# ---- Real-time ingestion pipeline (different service, different developer) ----

def upsert_realtime_batch(events: list[dict]) -> None:
    """
    events: list of {"id": str, "vector": list[float],
                     "timestamp": int, "category": str}
    timestamp stored as Unix integer: 1726401600
    """
    ns.upsert(
        ids=[e["id"] for e in events],
        vectors=[e["vector"] for e in events],
        attributes={
            "timestamp": [e["timestamp"] for e in events],
            # e.g. 1726401600
            # Schema already locked as text from first batch
            # Turbopuffer coerces integer to string "1726401600" silently
            # No error raised
            "category": [e["category"] for e in events],
        },
        distance_metric="cosine_distance",
    )
    # Upsert succeeds; no error raised
    # stored timestamp for real-time docs: "1726401600" (string)
    # stored timestamp for historical docs: "2026-09-15T12:00:00Z" (string)

# ---- Search function (broken) ----

def search_by_date_range(
    query_vector: list[float],
    start_ts: int,
    end_ts: int,
    top_k: int = 10,
) -> list:
    results = ns.query(
        vector=query_vector,
        top_k=top_k,
        distance_metric="cosine_distance",
        filters=["And", [
            ["timestamp", "Gte", start_ts],   # numeric $gte against text column
            ["timestamp", "Lte", end_ts],     # numeric $lte against text column
        ]],
    )
    # Returns zero results for real-time events
    # Historical ISO string timestamps also fail numeric comparison
    # No error — query succeeds with empty result set
    return results

Why Turbopuffer schema inference locks types at first upsert

Turbopuffer infers the schema of a namespace from the first batch of documents upserted into it. Each attribute's type — text, uint, bool, []text — is determined by the Python type of the values in that first batch. Once the schema is set, it is locked for the lifetime of the namespace. Subsequent upsert batches that provide attribute values of a different Python type are accepted without error, but the values are coerced to fit the locked schema type.

The coercion is the failure mode. When the schema has locked timestamp as text and the real-time pipeline sends integer 1726401600, Turbopuffer coerces the integer to its string representation "1726401600" and stores it. No exception is raised, and the upsert response gives no indication that type coercion occurred. The stored value is a string, indistinguishable in the stored data from a string that was explicitly provided as a string.

The filter failure follows directly. The production filter uses numeric operators Gte and Lte (corresponding to $gte and $lte) with integer arguments. When those operators are evaluated against a text-type column, the behavior diverges from numeric comparison. The ISO 8601 string "2026-09-15T12:00:00Z" does not match numeric comparisons against integers at all. The coerced string "1726401600" may match or not match depending on whether Turbopuffer applies lexicographic or type-error semantics — but in practice, the real-time documents are excluded from the results, and the developer never sees an error message that would point to the root cause.

# Diagnosing the schema type-lock trap — turbopuffer namespace inspection

import turbopuffer as tpuf

ns = tpuf.Namespace("fast_search_events")

# Step 1: inspect the namespace schema
schema = ns.schema()
print("Namespace schema:", schema)
# Namespace schema: {'timestamp': 'text', 'category': 'text'}
#                      ^^^^^^^^^ ← expected 'uint'; schema locked as 'text' by first batch

# Step 2: verify attribute values for a known real-time document
# Fetch by ID is not a direct Turbopuffer operation, but we can query by known ID
result = ns.query(
    vector=[0.0] * 1536,  # dummy vector
    top_k=1,
    filters=["id", "Eq", "realtime-doc-00001"],
    distance_metric="cosine_distance",
    include_attributes=True,
)
if result:
    print("timestamp attribute:", result[0].attributes["timestamp"])
    print("type:", type(result[0].attributes["timestamp"]))
    # timestamp attribute: 1726401600
    # type:    ← stored as string, not int

# Step 3: verify filter behavior
# Search with string comparison for a known coerced timestamp
result_str = ns.query(
    vector=[0.0] * 1536,
    top_k=5,
    filters=["timestamp", "Eq", "1726401600"],  # string equality
    distance_metric="cosine_distance",
)
print(f"String equality filter results: {len(result_str)}")
# String equality filter results: 1   ← document found with string comparison

result_num = ns.query(
    vector=[0.0] * 1536,
    top_k=5,
    filters=["timestamp", "Gte", 1726401600],   # numeric comparison
    distance_metric="cosine_distance",
)
print(f"Numeric Gte filter results: {len(result_num)}")
# Numeric Gte filter results: 0   ← confirms numeric filter fails on text column

The fix: delete the namespace, re-ingest with consistent integer timestamps

Because Turbopuffer namespace schemas are immutable once set, the only remedy is to delete the namespace and re-ingest all documents with a consistent attribute type. Both code paths — the historical backfill and the real-time pipeline — must store timestamp as an integer Unix timestamp. After re-ingestion, a pre-upsert metadata schema validator and a startup ns.schema() assertion prevent the same issue from recurring.

# search_client.py — FIXED
# turbopuffer Python SDK latest, Python 3.11, Turbopuffer Cloud
# All timestamps stored as Unix integers

import turbopuffer as tpuf
from datetime import datetime, timezone

ns = tpuf.Namespace("fast_search_events")

EXPECTED_SCHEMA = {
    "timestamp": "uint",
    "category": "text",
}

def assert_namespace_schema() -> None:
    """Fail fast if namespace schema does not match expected types."""
    actual = ns.schema()
    for attr, expected_type in EXPECTED_SCHEMA.items():
        actual_type = actual.get(attr)
        if actual_type != expected_type:
            raise RuntimeError(
                f"Namespace schema mismatch for attribute '{attr}': "
                f"expected '{expected_type}', got '{actual_type}'. "
                "If schema is locked as wrong type, delete the namespace and re-ingest "
                "with consistent attribute types. Schema is set at first upsert and is "
                "immutable for the lifetime of the namespace."
            )
    print("[OK] Namespace schema matches expected:", actual)

def validate_doc_schema(doc: dict) -> None:
    """Validate a single document's attribute types before upsert."""
    ts = doc.get("timestamp")
    if not isinstance(ts, int):
        raise TypeError(
            f"Document {doc.get('id')}: timestamp must be Unix integer, got {type(ts).__name__} "
            f"value {ts!r}. Convert ISO strings with: "
            "int(datetime.fromisoformat(ts.replace('Z', '+00:00')).timestamp())"
        )

def upsert_batch_validated(docs: list[dict]) -> None:
    """Validate attribute types then upsert to Turbopuffer namespace."""
    for doc in docs:
        validate_doc_schema(doc)

    ns.upsert(
        ids=[d["id"] for d in docs],
        vectors=[d["vector"] for d in docs],
        attributes={
            "timestamp": [d["timestamp"] for d in docs],
            # All integers after validation
            "category": [d["category"] for d in docs],
        },
        distance_metric="cosine_distance",
    )

def iso_to_unix(iso_str: str) -> int:
    """Convert ISO 8601 string to Unix integer timestamp."""
    return int(datetime.fromisoformat(iso_str.replace("Z", "+00:00")).timestamp())

def search_by_date_range(
    query_vector: list[float],
    start_ts: int,
    end_ts: int,
    top_k: int = 10,
) -> list:
    results = ns.query(
        vector=query_vector,
        top_k=top_k,
        distance_metric="cosine_distance",
        filters=["And", [
            ["timestamp", "Gte", start_ts],   # numeric — works with uint schema
            ["timestamp", "Lte", end_ts],
        ]],
    )
    # After fix: returns documents from both historical and real-time batches
    return results

# Call at application startup
assert_namespace_schema()

Re-ingestion procedure for 100,000 documents

# Re-ingestion procedure — Turbopuffer namespace reset

import turbopuffer as tpuf
from datetime import datetime, timezone

# Step 1: delete the namespace (drops schema lock and all stored documents)
ns = tpuf.Namespace("fast_search_events")
ns.delete_all_indexes()  # or ns.delete_all() depending on SDK version
print("Namespace cleared. Schema lock removed.")

# Step 2: re-ingest historical batch with converted timestamps
# Historical source stored timestamps as ISO 8601 strings — convert to integers

def iso_to_unix(iso_str: str) -> int:
    return int(datetime.fromisoformat(iso_str.replace("Z", "+00:00")).timestamp())

from your_database import get_historical_events, get_realtime_events

historical_events = get_historical_events()   # 50,000 docs with string timestamps
for doc in historical_events:
    doc["timestamp"] = iso_to_unix(doc["timestamp"])  # "2026-09-15T12:00:00Z" → 1726401600

# Upsert historical batch (first batch — sets schema as uint for timestamp)
BATCH_SIZE = 1000
for i in range(0, len(historical_events), BATCH_SIZE):
    batch = historical_events[i:i + BATCH_SIZE]
    upsert_batch_validated(batch)
    print(f"Historical: {min(i + BATCH_SIZE, len(historical_events))}/50000")

# Verify schema was inferred as uint
assert_namespace_schema()
# [OK] Namespace schema matches expected: {'timestamp': 'uint', 'category': 'text'}

# Step 3: re-ingest real-time batch (timestamps already integers — no conversion needed)
realtime_events = get_realtime_events()   # 50,000 docs with integer timestamps
for i in range(0, len(realtime_events), BATCH_SIZE):
    batch = realtime_events[i:i + BATCH_SIZE]
    upsert_batch_validated(batch)
    print(f"Realtime: {min(i + BATCH_SIZE, len(realtime_events))}/50000")

print("Re-ingestion complete. All 100,000 documents with consistent uint timestamps.")

Validation

# Validation log — November 2026
# turbopuffer Python SDK latest, Python 3.11, Turbopuffer Cloud

# ---- Before fix ----
# Namespace schema: {'timestamp': 'text', 'category': 'text'}
# Historical batch timestamps: ISO 8601 strings "2026-09-15T12:00:00Z"
# Real-time batch timestamps: Unix integers 1726401600 coerced to "1726401600"
# Date-range filter (Gte/Lte on timestamp): 0 results for all queries
# No errors raised anywhere in the stack

# ---- After fix: delete namespace + re-ingest with integer timestamps ----
# ns.schema(): {'timestamp': 'uint', 'category': 'text'}   ✓
# assert_namespace_schema(): [OK] Namespace schema matches expected              ✓
# Date-range filter (Gte/Lte on timestamp): returns correct documents from both batches ✓
# Total documents in namespace: 100,000                                          ✓
# Sample date-range query (Sept 2026): 847 results (correct)                    ✓

# ---- Hour breakdown ----
# Triage:     0.5h — date-range filtered queries return zero results;
#             queries without filter return correct documents;
#             ns.schema() shows timestamp as 'text'
# Root cause: 0.25h — compare first batch code path (ISO string) vs second (integer);
#             schema locked as text by first batch; integer coerced silently
# Fix:        0.5h — delete namespace; convert historical ISO strings to integers;
#             re-ingest 100,000 documents; pre-upsert validator; startup schema assertion
# Validation: 0.25h — schema shows uint; date-range filters return results; 100k docs present
# Total:      1.5h

# Retainer engagement cost: 1.5h (estimated $105–165 at mid-level rates)

Related retainer posts

Track retainer hours without a client portal

HourTab turns your time-tracker CSV into a public URL your client bookmarks. No login. No portal. CSV in, URL out.

See pricing →