Braintrust developer on retainer: asyncio.gather return_exceptions=True returns Exception objects — ZeroDivisionError when all LLM judge sub-evaluations time out — 20% of eval rows missing coherence scores for 4 weeks — Braintrust developer on monthly retainer

braintrust 0.0.178 • Python 3.11 • OpenAI GPT-4o • asyncio.gather return_exceptions=True • httpx.ReadTimeout from external scoring API • numeric=[] when all 3 sub-evals timeout • ZeroDivisionError at sum([]) / len([]) • Braintrust logs null coherence_score • 20 rows/week missing scores over 4 weeks • fix: empty-list guard + explicit httpx timeout • 100% coverage post-fix • 2.0h retainer engagement

Kestrel AI Solutions Inc. builds a code generation assistant that produces Python and TypeScript functions from natural-language specifications. They run a weekly automated eval using Braintrust to track the quality of the assistant’s output across a fixed 100-item test dataset. The eval measures three dimensions of output quality: relevance (does the function address the specification?), structure (does the code follow proper patterns?), and coherence (a composite score across structure, logic correctness, and style). After four weekly eval runs — September 22, September 29, October 6, and October 13, 2026 — the team noticed that the coherence scores column in the Braintrust experiment dashboard was empty for approximately 20 rows in each run. The rows showed the task output (the generated code) but no coherence score. They could not tell whether coherence had improved or regressed over four weeks because a fifth of the dataset had no coherence data at all. They brought in a Braintrust retainer developer to find out why 20% of rows were consistently missing coherence scores.

Their Braintrust eval was structured with three scoring functions. The first two — relevance_score and structure_score — called Braintrust’s built-in auto-evals and produced complete scores for every row. The third, coherence_score, was a custom async function that called an external code evaluation API to assess three independent quality dimensions in parallel:

# evals/weekly_eval.py
# Kestrel AI Solutions Inc.
# braintrust 0.0.178, Python 3.11

import asyncio
import httpx
import braintrust
from braintrust import Eval, Score

EVALUATE_API_KEY = os.environ["EVALUATE_API_KEY"]

async def call_evaluate_api(code: str, rubric_id: str) -> float:
    """Call external code evaluation API for one rubric dimension."""
    async with httpx.AsyncClient() as client:  # Default timeout: 5s connect, 5s read
        resp = await client.post(
            "https://api.evaluate.ai/v1/score",
            json={"code": code, "rubric_id": rubric_id},
            headers={"Authorization": f"Bearer {EVALUATE_API_KEY}"},
        )
        resp.raise_for_status()
        return resp.json()["score"]  # float between 0.0 and 1.0


async def coherence_score(input, output, expected):
    """
    Composite coherence score: average of structure, logic, and style subscores.
    Runs all 3 sub-evaluations in parallel using asyncio.gather.
    """
    results = await asyncio.gather(
        call_evaluate_api(output, rubric_id="code-structure"),
        call_evaluate_api(output, rubric_id="logic-correctness"),
        call_evaluate_api(output, rubric_id="code-style"),
        return_exceptions=True,  # Don't cancel other tasks if one fails
    )

    # Filter to only numeric results (ignore any exceptions)
    numeric = [s for s in results if isinstance(s, (int, float))]

    # Average the successful sub-scores
    return Score(name="coherence", score=sum(numeric) / len(numeric))
    #                                                     ^^^^^^^^^
    # BUG: raises ZeroDivisionError when numeric is empty
    #      (all 3 sub-evals failed / timed out)

The evaluation pipeline ran on Wednesdays at 09:00 UTC. Each of the 100 test cases called the coherence_score function, which made three concurrent HTTP requests to api.evaluate.ai. The external API was generally fast — typical response times were 800ms to 1,200ms. But approximately 20% of the time, on specific eval rows with longer generated code outputs, the API read latency exceeded the default httpx.AsyncClient read timeout of 5 seconds. When that happened, all three parallel requests to the same API endpoint — code-structure, logic-correctness, and code-style — timed out simultaneously. Each call raised httpx.ReadTimeout. The asyncio.gather call with return_exceptions=True captured those exceptions instead of propagating them, returning [httpx.ReadTimeout(), httpx.ReadTimeout(), httpx.ReadTimeout()].

The isinstance(s, (int, float)) filter correctly identified all three elements as non-numeric and excluded them. The result was numeric = []. The final line — sum(numeric) / len(numeric) — evaluated to sum([]) / len([]), which is 0 / 0, which raises ZeroDivisionError. Braintrust’s Eval() machinery wraps each scoring function call with exception handling: if a scoring function raises an exception, the corresponding score for that row is recorded as null and execution continues. The row was logged with the generated code output intact, but with an empty coherence column. This happened for all 20 rows per weekly eval run where the external API timed out, consistently and reproducibly.

The retainer developer confirmed the mechanism with a minimal reproduction:

# diagnosis/reproduce_zero_division.py
import asyncio
import httpx

async def simulate_all_timeouts():
    """Simulate what happens when all 3 sub-evals time out."""

    # Simulate 3 timeout exceptions (as if all sub-evals hit 5s read limit)
    results = await asyncio.gather(
        asyncio.sleep(0, result=httpx.ReadTimeout("timeout", request=None)),
        asyncio.sleep(0, result=httpx.ReadTimeout("timeout", request=None)),
        asyncio.sleep(0, result=httpx.ReadTimeout("timeout", request=None)),
        return_exceptions=False,  # doesn't matter here — already exceptions
    )

    # The actual path: simulate gather returning exception instances
    mocked_results = [
        httpx.ReadTimeout("timeout", request=None),
        httpx.ReadTimeout("timeout", request=None),
        httpx.ReadTimeout("timeout", request=None),
    ]

    numeric = [s for s in mocked_results if isinstance(s, (int, float))]
    print(f"numeric: {numeric}")  # []

    try:
        score = sum(numeric) / len(numeric)
    except ZeroDivisionError as e:
        print(f"ZeroDivisionError: {e}")  # ← This is what Braintrust catches

asyncio.run(simulate_all_timeouts())

# Output:
# numeric: []
# ZeroDivisionError: division by zero

# Next: confirmed by checking Braintrust eval results API for rows with null coherence
# braintrust.Eval() records null for this row's coherence score
# Row still visible in dashboard with task output but empty coherence column

The retainer developer also verified the API timeout frequency by pulling the eval run logs and counting rows by output code length. The pattern was clear: rows where the generated code was longer than approximately 2,000 characters triggered API read latency above 5 seconds, causing all three parallel calls to time out. The 20% timeout rate matched the proportion of test cases that generated outputs above that length threshold. The external API was processing longer inputs correctly — it just needed more than 5 seconds to return a score. The default httpx.AsyncClient timeout of 5 seconds was too aggressive for this use case.

The fix addressed both the immediate ZeroDivisionError and the underlying timeout configuration:

# evals/weekly_eval.py — fixed version
import asyncio
import httpx
import os
from braintrust import Eval, Score

EVALUATE_API_KEY = os.environ["EVALUATE_API_KEY"]

# Explicit timeout: 5s connect (unchanged), 30s read (was 5s)
EVALUATE_TIMEOUT = httpx.Timeout(connect=5.0, read=30.0, write=5.0, pool=5.0)

async def call_evaluate_api(code: str, rubric_id: str) -> float:
    """Call external code evaluation API with an explicit 30s read timeout."""
    async with httpx.AsyncClient(timeout=EVALUATE_TIMEOUT) as client:
        resp = await client.post(
            "https://api.evaluate.ai/v1/score",
            json={"code": code, "rubric_id": rubric_id},
            headers={"Authorization": f"Bearer {EVALUATE_API_KEY}"},
        )
        resp.raise_for_status()
        return resp.json()["score"]


async def coherence_score(input, output, expected):
    """
    Composite coherence score: average of structure, logic, and style subscores.
    Handles the case where all sub-evaluations fail (returns Score with score=0.0
    and metadata explaining the failure, rather than raising ZeroDivisionError).
    """
    results = await asyncio.gather(
        call_evaluate_api(output, rubric_id="code-structure"),
        call_evaluate_api(output, rubric_id="logic-correctness"),
        call_evaluate_api(output, rubric_id="code-style"),
        return_exceptions=True,
    )

    numeric = [s for s in results if isinstance(s, (int, float))]
    exceptions = [s for s in results if isinstance(s, Exception)]

    if not numeric:
        # All sub-evaluations failed — return a defined sentinel score
        # rather than raising ZeroDivisionError (which produces null in Braintrust)
        return Score(
            name="coherence",
            score=0.0,
            metadata={
                "error": "all_sub_evaluations_failed",
                "exception_types": [type(e).__name__ for e in exceptions],
                "exception_messages": [str(e) for e in exceptions],
            },
        )

    if exceptions:
        # Partial failure — log metadata but still return an average of successes
        avg_score = sum(numeric) / len(numeric)
        return Score(
            name="coherence",
            score=avg_score,
            metadata={
                "partial_failure": True,
                "succeeded": len(numeric),
                "failed": len(exceptions),
                "exception_types": [type(e).__name__ for e in exceptions],
            },
        )

    return Score(name="coherence", score=sum(numeric) / len(numeric))

After deploying the fix, the weekly eval run on October 20, 2026 showed 100% coherence score coverage for the first time: all 100 rows had a coherence score. Rows with longer outputs that had previously timed out now returned successfully within the 30-second window. No rows hit the new 30-second timeout — the longest API response in that run was 8.3 seconds, well within the 30-second limit. The four-week baseline was reconstructed by re-running the three prior eval runs against the existing dataset versions in Braintrust, this time with the fixed scoring function, producing complete coherence score history for all 400 rows.

The baseline analysis yielded a result the team had not expected: coherence scores had been improving steadily over the four weeks, from a mean of 0.71 on September 22 to 0.79 on October 13. This trend had been completely invisible while 20% of the data was missing. The product team had interpreted the absence of coherence data as a sign that the evaluation infrastructure was unreliable, not that the underlying model quality was improving. The retainer engagement produced both the fix and retroactive evidence of progress that had been accumulating undetected.

The postmortem documented three lessons. First: when using asyncio.gather with return_exceptions=True, any numeric aggregation downstream of the filter step must include an explicit empty-list guard. The guard belongs in the scoring function implementation itself, not in caller code, because Braintrust’s eval runner calls scoring functions independently. Second: HTTP client timeouts in scoring functions must be set explicitly. Library-default timeouts (5 seconds for httpx.AsyncClient read) are calibrated for general-purpose API calls; external LLM judge APIs that process variable-length inputs require longer read timeouts than the default. Third: null scores in Braintrust are silent. The Braintrust dashboard shows a — in the score column for null rows, which looks like a UI formatting choice rather than a data quality problem. The team added a post-eval check to their CI pipeline that reads the Braintrust experiment results via the API and fails the build if any scoring column has a null rate above 2%.

Total retainer engagement: 2.0 hours (0.5h initial triage: review four Braintrust experiment runs, confirm coherence score null pattern across all four weeks, identify that relevance and structure scores are complete while coherence is 20% null, focus investigation on coherence_score function; 0.5h root cause: read coherence_score implementation, identify asyncio.gather return_exceptions=True pattern, trace ZeroDivisionError path through empty numeric list, confirm with minimal reproduction script and API timeout frequency analysis by output length; 0.5h fix: update coherence_score with empty-list guard and explicit httpx.Timeout, update call_evaluate_api timeout, add partial-failure metadata to partial success case; 0.5h validation: deploy fix, run October 20 eval, confirm 100% coherence coverage, re-run prior weeks to reconstruct baseline trend).

Related retainer posts

• Opik developer on retainer — OPIK_PROJECT_NAME not set merges dev and prod traces, 68% correctness vs 88% production-only
• Humanloop developer on retainer — startup-cached prompt version not refreshed after Humanloop update, 12 days of stale production prompt
• LangSmith developer on retainer — single LANGCHAIN_PROJECT mixes staging and production traces, 4.2% reported error rate vs 1.1% actual