Qdrant developer on retainer: 489 unmerged segments cause scroll() with vectors to take 5.9 hours instead of 22 minutes — qdrant-client 1.9.0 — collection optimization threshold set to 50,000 never triggers on real-time ingest pattern — Qdrant developer on monthly retainer

qdrant-client 1.9.0 • Python 3.11 • FastAPI 0.111.0 • qdrant/qdrant:v1.9.0 Docker • text-embedding-3-small 1536 dims • 487,293 legal contract vectors • COSINE distance • NVMe SSD /data/qdrant • nightly S3 backup via scroll() with_vectors=True • indexing_threshold=50000 • 489 dirty segments, median 997 vectors each • 43 seconds per scroll page • 5.9 hours at month 8 • fix: indexing_threshold=0, 3 segments, 4 minutes • switch to snapshot API: 89 seconds • 3.0h retainer engagement

Meridian Document Intelligence Ltd. runs a SaaS semantic search platform for legal contracts. Freelancers upload contracts through a document processing pipeline; the service embeds each document using OpenAI’s text-embedding-3-small (1536 dimensions) and stores the vectors in a Qdrant collection called legal_contracts. Clients search approximately 2,000 times per day, returning similar clauses from the 487,293 stored vectors using cosine similarity. Every night at 2:00 AM UTC, a backup job exports the entire collection to S3 by scrolling through all points in pages of 1,000, serialising each page as JSON, and uploading it to a versioned S3 prefix. When the team first deployed the service in February 2026, the backup completed in 22 minutes. By October 2026 — eight months later — the backup was taking 5.9 hours and had begun timing out before the 6-hour CI/CD pipeline limit. Nothing obvious had changed. The Qdrant container was healthy, the NVMe disk was healthy, and S3 upload speeds were normal. The team had upgraded from qdrant 1.8.x to 1.9.0 in September with no improvement. A retainer developer was brought in when the backup began timing out intermittently and the team could not determine why the job was getting slower every month.

The backup service was straightforward: scroll through the collection with with_vectors=True, serialise each page, upload to S3. The code had not changed since the service launched:

# backup_service.py
# Meridian Document Intelligence Ltd.
# qdrant-client 1.9.0, Python 3.11

import asyncio
from qdrant_client import QdrantClient, models
import boto3
import json
import time

client = QdrantClient(url="http://localhost:6333")
s3 = boto3.client("s3")

async def backup_collection(collection_name: str, bucket: str):
    """Export all vectors to S3 for nightly backup."""
    offset = None
    page = 0
    total_points = 0

    start_time = time.time()

    while True:
        # BUG: scroll() with with_vectors=True performs random I/O across 489 segment files
        # Page 1: 22 seconds. Page 50: 29 seconds. Page 200: 41 seconds (I/O pressure compounds).
        results, next_offset = client.scroll(
            collection_name=collection_name,
            scroll_filter=None,
            limit=1000,
            with_payload=True,
            with_vectors=True,  # BUG: requires random disk seek per vector across fragmented segments
            offset=offset,
        )

        if not results:
            break

        # Serialize and upload page to S3
        page_data = [
            {
                "id": str(p.id),
                "vector": p.vector,
                "payload": p.payload
            }
            for p in results
        ]
        s3.put_object(
            Bucket=bucket,
            Key=f"backups/{collection_name}/page_{page:04d}.json",
            Body=json.dumps(page_data)
        )

        total_points += len(results)
        page += 1
        offset = next_offset

        if next_offset is None:
            break

    elapsed = time.time() - start_time
    print(f"Backup complete: {total_points} points in {elapsed:.1f}s across {page} pages")
    return total_points

The backup job had produced consistent timing data over its eight months of operation. The team had logged job duration in their deployment records and the trend was visible in hindsight:

• Month 1 (February): 3 pages × 22 seconds = 66 seconds ≈ 1 minute
• Month 4 (May): 122 pages × 28 seconds = 57 minutes
• Month 8 (October): 488 pages × 43 seconds = 5.9 hours

The team had tried adjusting the page size in October when the timeouts began. Reducing the page size from 1,000 to 100 made things dramatically worse: 4,880 pages × 43 seconds each would have been 58 hours if they had let it run. Increasing the page size to 5,000 helped slightly — 98 pages × 180 seconds = 4.9 hours — but the core problem remained. The team correctly observed that the issue was not S3 bandwidth (network throughput was fine; the bottleneck was before the upload step) and not CPU or RAM inside the Qdrant container (both were low during the backup). NVMe SMART data showed a healthy drive. The problem was the number of disk seeks, not their speed.

Diagnosis: collection stats reveal the root cause

The retainer developer’s first action was to call get_collection() and compare three numbers: points_count, indexed_vectors_count, and segments_count:

# Diagnostic: check collection stats
info = client.get_collection("legal_contracts")
print(f"Points: {info.points_count}")                    # 487,293
print(f"Indexed vectors: {info.indexed_vectors_count}")  # 31,847 (only 3 original large segments indexed)
print(f"Segments: {info.segments_count}")                # 489

# Benchmark a single scroll page
import time
start = time.time()
results, _ = client.scroll(
    collection_name="legal_contracts",
    limit=1000,
    with_vectors=True,
    offset=None
)
print(f"Single page (1,000 vectors): {time.time() - start:.1f}s")  # 43.1s

# Compare: scroll WITHOUT vectors (payload only)
start = time.time()
results, _ = client.scroll(
    collection_name="legal_contracts",
    limit=1000,
    with_vectors=False,   # payload only -- sequential index scan
    offset=None
)
print(f"Single page without vectors: {time.time() - start:.1f}s")  # 0.08s -- 538x faster

The numbers told the story immediately. points_count was 487,293 but indexed_vectors_count was only 31,847 — 6.5% of the total. The collection had 489 segments. A healthy Qdrant collection with 487,293 vectors should have 2 to 5 segments and 100% of vectors indexed. This collection had 489 segments and 93.5% of its vectors unindexed. The benchmark confirmed the mechanism: scrolling with with_vectors=True took 43.1 seconds per page; scrolling with with_vectors=False took 0.08 seconds per page — a 538x difference. The payload-only scroll was fast because Qdrant uses a sequential scan of the point ID index, which is stored in a compact, memory-friendly structure. The vector-including scroll was slow because Qdrant had to locate each of the 1,000 vectors on disk, and with 489 segment files, each page required reads scattered across 489 separate files on NVMe. Random I/O.

The next step was to understand why 489 segments existed. The retainer developer reviewed the ingest code:

# ingest_service.py
# Meridian Document Intelligence Ltd.
# Called from document processing pipeline webhook (FastAPI endpoint)

from qdrant_client import QdrantClient, models
import numpy as np

client = QdrantClient(url="http://localhost:6333")

def ingest_document_batch(embeddings: list[np.ndarray], payloads: list[dict], ids: list[str]):
    """Ingest a batch of document embeddings into Qdrant."""
    points = [
        models.PointStruct(
            id=ids[i],
            vector=embeddings[i].tolist(),
            payload=payloads[i]
        )
        for i in range(len(embeddings))
    ]

    # wait=False: returns immediately, Qdrant processes asynchronously.
    # Each call creates or appends to a dirty (unindexed) segment.
    # The optimizer only builds HNSW when a dirty segment reaches indexing_threshold.
    client.upsert(
        collection_name="legal_contracts",
        points=points,
        wait=False,   # asynchronous -- fast for real-time ingest
    )
    return len(points)

The ingest batches ranged from 200 to 500 points depending on document size. With ~800 new documents per day arriving via webhooks, the ingest created 2 to 4 upsert calls per day, each creating or extending a dirty segment. The key question was: why did those dirty segments never get optimized and merged into indexed segments?

The answer was in the Qdrant optimizer configuration. The retainer developer found the configuration history in the team’s Git repository. In week 3 of the deployment, a developer had changed indexing_threshold from the Qdrant default of 20,000 to 50,000 after reading a blog post that recommended “reducing optimizer overhead by increasing the indexing threshold.” The recommendation was intended for bulk-ingest pipelines where large batches of 5,000 to 20,000 vectors arrive together — in that pattern, a segment can grow to 50,000 vectors relatively quickly and the optimizer fires efficiently. For Meridian’s real-time pattern of 200- to 500-point batches, a segment could accumulate for months and never approach 50,000 vectors before new batches created new dirty segments alongside it. The optimizer had never once triggered for any of the real-time segments. The only indexed segments (the 31,847 vectors) were from the initial bulk import done at launch before the threshold was changed.

The math was straightforward. Each NVMe random read takes approximately 80–100 microseconds. A scroll page of 1,000 vectors requires 1,000 vector lookups. If all 487,293 vectors are in 3 indexed segments, the vectors are stored contiguously and the 1,000 lookups can be served from a small number of sequential reads — fast. If the 487,293 vectors are distributed across 489 dirty segment files, each lookup potentially touches a different file. The NVMe queue depth (typically 32–64) limits parallelism; at ~12,000 random read IOPS, a page of 1,000 vector lookups scattered across 489 files takes 1,000 ÷ 12,000 × the number of round trips ≈ 40–45 seconds. This matched the observed 43.1 seconds per page precisely.

Fix: force optimization then switch to snapshot backups

The fix had two parts: immediately merge all dirty segments by setting indexing_threshold=0, then prevent the problem from recurring by setting a production threshold appropriate for the ingest batch size, and replace the scroll-based backup with the Qdrant snapshot API.

# Step 1: Force collection optimization to merge all segments
client.update_collection(
    collection_name="legal_contracts",
    optimizer_config=models.OptimizersConfigDiff(
        indexing_threshold=0,           # Build HNSW immediately for any segment size
        deleted_threshold=0.2,          # Merge segments with >20% deleted vectors
        vacuum_min_vector_number=1000,
    )
)

# Step 2: Wait for optimization to complete (may take 15-30 minutes)
import time
while True:
    info = client.get_collection("legal_contracts")
    status = info.optimizer_status
    print(f"Optimizer status: {status}, segments: {info.segments_count}")
    if hasattr(status, 'ok') and status.ok:
        break
    time.sleep(30)

# After optimization: segments reduced from 489 to 3, indexed_vectors_count = 487,293
# Post-optimization scroll benchmark:
start = time.time()
results, _ = client.scroll(
    collection_name="legal_contracts",
    limit=1000,
    with_vectors=True,
    offset=None
)
print(f"Single page after optimization: {time.time() - start:.1f}s")  # 0.51s
# 488 pages x 0.51s = 4.1 minutes total -- vs previous 5.9 hours

The optimization ran for 22 minutes. During that window, the Qdrant container’s CPU usage rose to 60–70% as the HNSW optimizer worked through all 489 dirty segments, building indexes and merging them. After optimizer_status.ok became True, the collection stats showed exactly what was expected: segments_count=3, indexed_vectors_count=487,293 (100%), points_count=487,293. The scroll benchmark dropped from 43.1 seconds per page to 0.51 seconds per page. A full 488-page scroll of the collection now completed in 4.1 minutes — a 86x improvement, and well within the backup window.

However, even at 4.1 minutes, scroll-based JSON export is not the right backup strategy for a production Qdrant collection. The retainer developer recommended switching to the Qdrant snapshot API, which creates a binary snapshot of the entire collection — including the HNSW index structure, all vectors, and all payloads — as a single file. For the legal_contracts collection (487,293 points, 1536 dims), a snapshot creation took 89 seconds. The snapshot is also a complete restoration artifact: restoring from snapshot recreates the collection with the exact same segment structure, indexes, and optimizer state, without requiring Qdrant to rebuild HNSW from scratch. A scroll-based JSON export, by contrast, requires re-ingesting all 487,293 vectors to restore, which would trigger another optimization pass and take 30+ minutes to index.

# Better backup approach: Qdrant snapshot API
# Creates a binary snapshot of the entire collection in ~90 seconds
# Snapshot is a complete, restorable copy including HNSW index

import httpx

async def backup_collection_snapshot(collection_name: str, bucket: str):
    """Create Qdrant snapshot and upload to S3."""
    async with httpx.AsyncClient(timeout=300.0) as http:
        # Create snapshot -- returns immediately once snapshot file is written
        response = await http.post(
            f"http://localhost:6333/collections/{collection_name}/snapshots"
        )
        snapshot = response.json()["result"]
        snapshot_name = snapshot["name"]

        # Download and stream to S3 multipart upload
        s3_client = boto3.client("s3")
        mpu = s3_client.create_multipart_upload(
            Bucket=bucket,
            Key=f"snapshots/{collection_name}/{snapshot_name}"
        )
        parts = []
        part_number = 1

        async with http.stream(
            "GET",
            f"http://localhost:6333/collections/{collection_name}/snapshots/{snapshot_name}"
        ) as stream:
            buffer = b""
            async for chunk in stream.aiter_bytes(chunk_size=8 * 1024 * 1024):
                buffer += chunk
                if len(buffer) >= 50 * 1024 * 1024:  # 50MB parts
                    part = s3_client.upload_part(
                        Bucket=bucket,
                        Key=f"snapshots/{collection_name}/{snapshot_name}",
                        PartNumber=part_number,
                        UploadId=mpu["UploadId"],
                        Body=buffer,
                    )
                    parts.append({"PartNumber": part_number, "ETag": part["ETag"]})
                    part_number += 1
                    buffer = b""
            if buffer:
                part = s3_client.upload_part(
                    Bucket=bucket,
                    Key=f"snapshots/{collection_name}/{snapshot_name}",
                    PartNumber=part_number,
                    UploadId=mpu["UploadId"],
                    Body=buffer,
                )
                parts.append({"PartNumber": part_number, "ETag": part["ETag"]})

        s3_client.complete_multipart_upload(
            Bucket=bucket,
            Key=f"snapshots/{collection_name}/{snapshot_name}",
            UploadId=mpu["UploadId"],
            MultipartUpload={"Parts": parts},
        )

        # Delete snapshot from Qdrant local storage after upload
        await http.delete(
            f"http://localhost:6333/collections/{collection_name}/snapshots/{snapshot_name}"
        )

    return snapshot_name

The production configuration was also updated to prevent segment accumulation from recurring. The indexing_threshold was set to 10,000 — low enough that Meridian’s ingest batches of 200–500 vectors would trigger the optimizer after 20–50 batches (roughly one to two days of ingest), instead of never:

# qdrant config (config/production.yaml)
storage:
  optimizers:
    indexing_threshold: 10000    # Was: 50000; reduced to trigger HNSW on smaller segments
    deleted_threshold: 0.2
    vacuum_min_vector_number: 1000
    default_segment_number: 2
    max_segment_size: 100000

The team added a weekly monitoring job that calls get_collection() and alerts if segments_count exceeds 20 or if indexed_vectors_count / points_count drops below 85%. These thresholds would detect a recurrence of segment fragmentation long before it affected backup or search performance.

Why the team’s investigation missed the cause

The team had checked everything reasonable: disk health, container resource usage, network bandwidth, Qdrant version. All were normal. The check they had not done was get_collection() and comparing indexed_vectors_count to points_count. This is not a commonly documented diagnostic step — most Qdrant troubleshooting guides focus on search recall (which is also affected by unindexed segments, but less dramatically) rather than scroll performance. The 538x difference between with_vectors=True and with_vectors=False scroll benchmarks was the key diagnostic signal; without running that comparison, the root cause was invisible in all standard monitoring metrics.

The team had also tried the intuitive fix — reducing page size from 1,000 to 100 — which made things dramatically worse. This is counterintuitive: smaller pages should mean less work per page. But the per-page I/O time was dominated by the number of segment files read (up to 489), not the number of vectors per page. Reducing the page size from 1,000 to 100 reduced vectors per page by 10x but increased page count by 10x, so total I/O remained constant and the overhead per page (establishing file handles for up to 489 segment files) now repeated 10x as often. The result was 58 hours instead of 5.9 hours. The only page-size tuning that helped was increasing to 5,000, which reduced the number of times the per-page setup cost was paid — but the fundamental random I/O problem was not addressable by page size tuning.

The retainer developer also noted that the wait=False parameter in the ingest code was appropriate for Meridian’s use case. Switching to wait=True would have increased ingest latency significantly — each upsert call would block until Qdrant confirmed the segment was written, adding 50–200 ms per batch to the document processing webhook response time. The right fix was not to change the ingest pattern but to configure the optimizer correctly for that pattern. With indexing_threshold=10,000, the optimizer now fires automatically after a dirty segment accumulates enough vectors, keeping the segment count low without requiring any change to the ingest code.

Total retainer engagement: 3.0 hours (0.5h triage: call get_collection(); record segments_count=489, indexed_vectors_count=31,847; benchmark scroll(with_vectors=True) at 43.1 seconds; benchmark scroll(with_vectors=False) at 0.08 seconds; confirm 538x ratio points to segment fragmentation and random I/O as root cause; 1.0h root cause: review ingest_service.py; confirm wait=False and batches of 200–500; review Git history for config/production.yaml; find indexing_threshold=50000 change in week 3; confirm batches never reach 50,000 vectors; calculate 8 months × 30 days × 3 segments/day = ~489 surviving segments; confirm all 489 are unindexed dirty segments; confirm 31,847 indexed vectors are only from the initial bulk import; 1.0h fix: call update_collection(indexing_threshold=0); wait 22 minutes for optimizer; confirm segments_count=3, indexed_vectors_count=487,293; run scroll benchmark at 0.51 seconds/page; 488 pages × 0.51s = 4.1 minutes; implement snapshot-based backup; run backup in staging: 89 seconds; 0.5h validation: deploy production config with indexing_threshold=10000; confirm optimizer runs as new batches arrive; add weekly monitoring alert on segments_count > 20; confirm nightly backup completes in 89 seconds; monitor for 1 week).

Related retainer posts

• Tecton developer on retainer — feature store retrieval performance
• Hopsworks developer on retainer — vector store configuration
• BentoML developer on retainer — model serving infrastructure