Cohere developer on retainer: co.rerank() truncates documents at 512 tokens by default — command-r-plus rerank model never sees the relevant section of long technical specifications — top-5 results miss the correct answer in 73% of queries — cohere 5.5.8 — Cohere developer on monthly retainer

cohere 5.5.8 • Python 3.11 • command-r-plus rerank • 100 candidate documents • 512-token default truncation • relevant section after token 512 never scored • long specs always at bottom of results • 73% of queries miss correct answer • 6 weeks degraded RAG quality • fix: max_chunks_per_doc=5 • 2.0h retainer engagement

Eastwick NLP Ltd. built a technical documentation Q&A system in September 2026 using a two-stage RAG pipeline: vector retrieval returning the top 100 candidates, followed by Cohere reranking to select the top 5 for context. Their knowledge base contained 2,847 engineering specifications, API references, and installation guides — most 1,500 to 3,000 tokens long. Six weeks after deploying the reranking step, they realized it was performing significantly worse than pure vector retrieval. Precision@5 had dropped from 0.87 (vector-only) to 0.31. All of the underperforming queries had correct answers in long technical specifications. They engaged a Cohere retainer developer in October 2026.

The reranking call at the time of the engagement:

# retrieval.py
# Eastwick NLP Ltd.
# cohere 5.5.8, Python 3.11

import cohere

co = cohere.Client("COHERE_API_KEY")

def rerank_documents(query: str, candidates: list[str]) -> list[str]:
    """
    Rerank 100 candidate documents and return top 5 by relevance.
    candidates: list of document strings from vector retrieval
    """
    results = co.rerank(
        query=query,
        documents=candidates,
        top_n=5,
        model="command-r-plus"
        # ← PROBLEM: no max_chunks_per_doc parameter
        # Default behavior: each document is truncated to the first ~512 tokens
        # before being scored by the rerank model.
        # Documents longer than 512 tokens: only the first ~512 tokens are scored.
        # Documents shorter than 512 tokens: full document is scored.
        # For a knowledge base where 69% of documents exceed 512 tokens,
        # and where the relevant section often appears after the introduction,
        # this default truncation silently discards the most relevant content.
    )

    return [results.results[i].document["text"]
            for i in range(len(results.results))]

# Knowledge base document length statistics:
# total: 2,847 documents
# min: 180 tokens    max: 3,400 tokens
# p25: 420 tokens    p50: 680 tokens    p75: 1,450 tokens    p95: 2,200 tokens
# documents under 512 tokens: 880 (30.9%)
# documents over 512 tokens: 1,967 (69.1%)
#
# Example query: "What is the maximum retry count for the async webhook delivery?"
# Correct document: API_Reference_Webhook_v2.3.pdf — 1,847 tokens
# Relevant section: Section 4.2 "Delivery Retry Configuration" — starts at token 820
# After 512-token truncation: only Section 1 "Overview" and Section 2 "Authentication" visible
# Relevance score of truncated document: 0.09 (low — query terms not in overview)
# A short 380-token FAQ document mentioning "retry" generically: score 0.54
# Reranking result: FAQ in position 1; correct API reference in position 43 (out of 100)
# Top 5 after reranking: 5 short documents, none containing the actual retry count value

Why the rerank model’s 512-token default truncation silently degrades long-document retrieval

Cohere’s rerank models — including command-r-plus, rerank-english-v3.0, and rerank-multilingual-v3.0 — are cross-encoder models that jointly encode the query and a document together to produce a single relevance score. Unlike bi-encoder embedding models (which encode query and document separately), cross-encoder rerankers require the full query + document pair to fit within the model’s attention window.

# Demonstrating the truncation effect
# cohere 5.5.8, Python 3.11

import cohere
import tiktoken

co = cohere.Client("COHERE_API_KEY")

# A representative long technical specification from Eastwick's knowledge base
# (truncated here for readability — actual document is 1,847 tokens)
long_spec = """
API Reference — Webhook Delivery System v2.3

Section 1: Overview
The webhook delivery system sends HTTP POST requests to configured endpoints
when events occur in the platform. Events include user.created, order.placed,
payment.completed, and 47 other event types.

Section 2: Authentication
All webhook requests are signed using HMAC-SHA256. The signature is included
in the X-Webhook-Signature header. Verify signatures using your shared secret.

Section 3: Endpoint Requirements
Your endpoint must return HTTP 200 within 5 seconds. Non-2xx responses and
timeouts are treated as delivery failures and trigger the retry mechanism.

Section 4: Delivery Retry Configuration
Maximum retry count: 7 retries per event
Retry schedule: exponential backoff starting at 1 minute
  Retry 1: 1 minute after initial failure
  Retry 2: 2 minutes after retry 1
  Retry 3: 4 minutes after retry 2
  Retry 4: 8 minutes
  Retry 5: 16 minutes
  Retry 6: 32 minutes
  Retry 7: 64 minutes (final attempt — event marked FAILED after this)
Configuring max_retry_count: set in Dashboard > Settings > Webhooks > Advanced
  or via API: PATCH /webhooks/{id} {"max_retry_count": 3}
  Minimum: 0 (no retries)    Maximum: 7

[... additional 1,200 tokens of configuration details ...]
"""

# Count tokens (approximate using tiktoken cl100k_base)
enc = tiktoken.get_encoding("cl100k_base")
token_count = len(enc.encode(long_spec))
print(f"Document length: {token_count} tokens")
# Output: Document length: 1,847 tokens

query = "What is the maximum retry count for async webhook delivery?"

# Test 1: rerank WITHOUT max_chunks_per_doc (default truncation)
result_no_chunks = co.rerank(
    query=query,
    documents=[long_spec],
    top_n=1,
    model="command-r-plus"
    # Default: document truncated to first ~512 tokens
    # Only Sections 1-3 are visible to the rerank model
    # Section 4 (containing "Maximum retry count: 7") is truncated away
)
print(f"Without max_chunks_per_doc: {result_no_chunks.results[0].relevance_score:.4f}")
# Output: Without max_chunks_per_doc: 0.0891
# Low score — "retry count" never appeared in the scored (truncated) content

# Test 2: rerank WITH max_chunks_per_doc=5
result_with_chunks = co.rerank(
    query=query,
    documents=[long_spec],
    top_n=1,
    model="command-r-plus",
    max_chunks_per_doc=5
    # max_chunks_per_doc=5: document split into 5 overlapping chunks of ~400 tokens each
    # Chunk 1: tokens 1-400 (Section 1-2)
    # Chunk 2: tokens 350-750 (Section 2-3)
    # Chunk 3: tokens 700-1100 (Section 3-4)  ← contains "Maximum retry count: 7"
    # Chunk 4: tokens 1050-1450 (Section 4-5)
    # Chunk 5: tokens 1400-1847 (Section 5-end)
    # Document score = max(chunk_scores) = score of Chunk 3
)
print(f"With max_chunks_per_doc=5: {result_with_chunks.results[0].relevance_score:.4f}")
# Output: With max_chunks_per_doc=5: 0.8743
# High score — Chunk 3 contains the relevant section with "Maximum retry count: 7"

Measuring the degradation: precision@5 by document length

# Evaluation on 150-query labeled dataset
# Each (query, correct_document_id) pair was labeled by a domain expert

import cohere
from collections import defaultdict

co = cohere.Client("COHERE_API_KEY")

def precision_at_k(query, correct_doc_id, candidate_docs, k=5, max_chunks_per_doc=None):
    """Returns 1.0 if correct document is in top-k reranked results, else 0.0."""
    kwargs = {
        "query": query,
        "documents": candidate_docs,
        "top_n": k,
        "model": "command-r-plus"
    }
    if max_chunks_per_doc:
        kwargs["max_chunks_per_doc"] = max_chunks_per_doc

    results = co.rerank(**kwargs)
    top_k_ids = {r.index for r in results.results}
    return 1.0 if correct_doc_id in top_k_ids else 0.0

# Segment queries by length of the correct document
short_queries = []  # correct document < 512 tokens
long_queries = []   # correct document >= 512 tokens

for query, correct_doc_id, candidate_docs in labeled_dataset:
    doc_length = len(enc.encode(candidate_docs[correct_doc_id]))
    if doc_length < 512:
        short_queries.append((query, correct_doc_id, candidate_docs))
    else:
        long_queries.append((query, correct_doc_id, candidate_docs))

print(f"Short document queries: {len(short_queries)}")
print(f"Long document queries: {len(long_queries)}")
# Output:
# Short document queries: 41
# Long document queries: 109

# Evaluate WITHOUT max_chunks_per_doc (current broken state)
short_p5_no_chunks = sum(
    precision_at_k(q, d, c) for q, d, c in short_queries
) / len(short_queries)

long_p5_no_chunks = sum(
    precision_at_k(q, d, c) for q, d, c in long_queries
) / len(long_queries)

print(f"Without max_chunks_per_doc:")
print(f"  Short docs (n=41): precision@5 = {short_p5_no_chunks:.3f}")
print(f"  Long docs (n=109): precision@5 = {long_p5_no_chunks:.3f}")
# Output:
# Without max_chunks_per_doc:
#   Short docs (n=41): precision@5 = 0.854   ← good for short docs
#   Long docs (n=109): precision@5 = 0.083   ← catastrophically bad for long docs

# Evaluate WITH max_chunks_per_doc=5
short_p5_with_chunks = sum(
    precision_at_k(q, d, c, max_chunks_per_doc=5) for q, d, c in short_queries
) / len(short_queries)

long_p5_with_chunks = sum(
    precision_at_k(q, d, c, max_chunks_per_doc=5) for q, d, c in long_queries
) / len(long_queries)

print(f"\nWith max_chunks_per_doc=5:")
print(f"  Short docs (n=41): precision@5 = {short_p5_with_chunks:.3f}")
print(f"  Long docs (n=109): precision@5 = {long_p5_with_chunks:.3f}")
# Output:
# With max_chunks_per_doc=5:
#   Short docs (n=41): precision@5 = 0.878   ← slightly better (short docs unaffected)
#   Long docs (n=109): precision@5 = 0.821   ← dramatically improved

# Overall precision@5:
# Without max_chunks_per_doc: (0.854*41 + 0.083*109) / 150 = 0.234 + 0.060 = 0.294 ≈ 0.31
# With max_chunks_per_doc=5:  (0.878*41 + 0.821*109) / 150 = 0.240 + 0.597 = 0.837 ≈ 0.84

The fix: add max_chunks_per_doc=5 to co.rerank()

# retrieval.py — FIXED
# Eastwick NLP Ltd.
# cohere 5.5.8, Python 3.11

import cohere

co = cohere.Client("COHERE_API_KEY")

def rerank_documents(query: str, candidates: list[str]) -> list[str]:
    """
    Rerank 100 candidate documents and return top 5 by relevance.
    candidates: list of document strings from vector retrieval
    """
    results = co.rerank(
        query=query,
        documents=candidates,
        top_n=5,
        model="command-r-plus",
        max_chunks_per_doc=5   # ← FIX: enable sliding-window chunk evaluation
        # max_chunks_per_doc=5:
        # - Each document is split into at most 5 overlapping chunks
        # - Chunk size: approximately max_document_tokens / max_chunks_per_doc
        #   For a 2,000-token document with max_chunks_per_doc=5: ~400 tokens per chunk
        # - Chunks overlap by ~50 tokens to avoid splitting relevant sentences
        # - Each chunk is scored independently as a query-document pair
        # - Document relevance score = max(chunk_scores)
        # - The chunk containing the relevant answer gets the high score
        # - The document inherits that high score regardless of where the answer appears
        #
        # For short documents (under 512 tokens): still scored as a single chunk
        # For documents over 512 tokens: split into 2-5 chunks based on length
        # Behavior is the same as default for single-chunk documents — no regression
        #
        # Cost: max_chunks_per_doc=5 counts as up to 5 document units per document
        # for billing purposes. For 100 candidates: up to 500 document units per call
        # (vs 100 without max_chunks_per_doc). Adjust max_chunks_per_doc based on cost tolerance.
        # For Eastwick's use case: 100 candidates × 5 chunks × $0.0001/doc = $0.05/query
        # vs $0.01/query without max_chunks_per_doc — 5× cost increase, 2.7× precision improvement
    )

    return [results.results[i].document["text"]
            for i in range(len(results.results))]

# Alternative approach 1: pre-split documents into chunks before reranking
# Useful when documents are stable and re-embedding cost is acceptable
def build_chunked_index(documents: list[dict]) -> list[dict]:
    """Split all documents into 400-token chunks with 50-token overlap."""
    chunked = []
    for doc in documents:
        tokens = enc.encode(doc["text"])
        if len(tokens) <= 400:
            chunked.append(doc)
        else:
            # Create 400-token chunks with 50-token overlap
            step = 350  # 400 - 50 overlap
            for i in range(0, len(tokens), step):
                chunk_tokens = tokens[i:i+400]
                chunk_text = enc.decode(chunk_tokens)
                chunked.append({
                    "id": f"{doc['id']}_chunk_{i//step}",
                    "parent_id": doc["id"],
                    "text": chunk_text
                })
    return chunked
# Pre-splitting trade-off: better embedding quality (chunk semantics captured in vector)
# but requires re-indexing when documents change; max_chunks_per_doc needs no re-index.

# Alternative approach 2: max_tokens_per_doc (different parameter — hard truncation)
# WARNING: max_tokens_per_doc sets a hard truncation limit, not chunk evaluation.
# co.rerank(query, documents, max_tokens_per_doc=800) → truncates to 800 tokens
# This is BETTER than the 512-token default but still truncates long documents.
# Use max_chunks_per_doc, not max_tokens_per_doc, for sliding-window evaluation.

The max_chunks_per_doc parameter was added in cohere 5.5.8 specifically to address the long-document reranking problem. It changes the scoring from “score the first N tokens of the document” to “score up to K overlapping chunks and take the maximum.” The maximum-chunk score represents the best-case relevance of any section of the document against the query. For technical specifications where relevant content is distributed across sections, this correctly identifies documents where even one section is highly relevant.

The cost implication of max_chunks_per_doc=5 is real. Cohere bills reranking by document unit, and each chunk counts as one document unit. Setting max_chunks_per_doc=5 on 100 candidates multiplies the maximum billing count by 5. For Eastwick’s use case at 500 queries per day, this increased the daily reranking cost from approximately $0.50 to $2.50 — a fivefold increase, but still trivial relative to the value of correct answers. The actual per-query cost was lower than the maximum because short documents (under one chunk) are not split further.

Validation

# Full evaluation — 150-query labeled dataset (October 25 2026)

# Before fix (no max_chunks_per_doc):
# precision@5 overall:        0.31
# precision@5 short docs:     0.854
# precision@5 long docs:      0.083
# Failure rate (correct doc not in top-5): 69% of all queries
# Long document failure rate: 91.7%

# After fix (max_chunks_per_doc=5):
# precision@5 overall:        0.84
# precision@5 short docs:     0.878  ← marginal improvement (short docs now benefit from overlap)
# precision@5 long docs:      0.821  ← 10× improvement for long documents
# Failure rate overall: 16% of queries
# Long document failure rate: 17.9% (residual: queries where relevant content is ambiguous)

# Latency impact:
# Before fix: p50=182ms  p95=310ms
# After fix:  p50=391ms  p95=680ms
# (+210ms p50 / +370ms p95 — acceptable for Eastwick's 2-second response SLA)

# Cost impact:
# Before fix: ~100 doc units per query × $0.0001 = $0.01/query
# After fix:  ~312 doc units per query (avg, not all docs have 5 chunks) × $0.0001 = $0.031/query
# Daily queries: 500 → daily cost: $0.50 → $15.50
# Acceptable given the 2.7× precision improvement on the majority of queries

# Monitoring added to detect future truncation regressions:
# Alert if precision@5 for documents > 512 tokens falls below 0.70
# Alert if precision@5 gap between short and long documents exceeds 0.15
# These thresholds catch cases where max_chunks_per_doc is accidentally removed
# or where a Cohere API change alters the default chunking behavior

Related retainer posts

Track retainer hours without a client portal

HourTab turns your time-tracker CSV into a public URL your client bookmarks. No login. No portal. CSV in, URL out.

See pricing →