Weave developer on retainer: LLM eval Scorer returns float but aggregation expects dict key “score” — weave 0.50.8 silently wraps scalar in {“output”: 0.82} — all eval results show NaN in dashboard — 4 weeks — Weave developer on monthly retainer

weave 0.50.8 • Python 3.11 • OpenAI 1.35.0 • weave.Evaluation • custom Scorer subclass • score() returns float • weave wraps scalar in {“output”: val} • aggregation expects key “score” • all eval results NaN for 4 weeks • no exception raised • fix: return {“score”: float} • 2.0h retainer engagement

Clearbrook ML Ops Ltd. built an LLM evaluation pipeline in September 2026 using Weights & Biases Weave to track quality across their RAG system. The pipeline ran 600 documents through their retrieval and generation stack, scored each response with a custom accuracy scorer, and reported results to the Weave dashboard. The evaluation runs completed with green status icons every time. But the dashboard showed NaN for every scorer column, every run, for four weeks. They engaged a Weave retainer developer to find out why.

The evaluation pipeline at the time of the engagement:

# eval_pipeline.py
# Clearbrook ML Ops Ltd.
# weave 0.50.8, Python 3.11

import weave
from weave import Scorer

weave.init("clearbrook-rag-eval")

class AccuracyScorer(Scorer):
    """Score LLM answer accuracy against ground-truth target."""

    @weave.op()
    def score(self, output: str, target: str) -> float:
        # ← PROBLEM: returns a plain float, not a dict
        # weave 0.50.8 Scorer contract: score() must return a dict.
        # When a scalar is returned, weave silently wraps it:
        #   {'output': 0.82}
        # The aggregation step then looks for key 'score', not 'output'.
        # Key 'score' is never present. Mean of empty series = NaN.
        # No exception. No warning. Evaluation runs succeed. Dashboard shows NaN.
        predicted = output.strip().lower()
        expected = target.strip().lower()
        if predicted == expected:
            return 1.0
        if expected in predicted or predicted in expected:
            return 0.5
        return 0.0

class GroundednessScorer(Scorer):
    """Score whether the answer is grounded in the retrieved context."""

    @weave.op()
    def score(self, output: str, context: str) -> float:
        # ← Same problem: returns float, not {'score': float}
        output_words = set(output.lower().split())
        context_words = set(context.lower().split())
        overlap = len(output_words & context_words)
        if not output_words:
            return 0.0
        return min(1.0, overlap / len(output_words))

# weave.Evaluation takes:
#   dataset: list of dicts (each dict is one eval row)
#   scorers: list of Scorer instances or callables
evaluation = weave.Evaluation(
    dataset=eval_dataset,     # 600 rows: {question, context, target}
    scorers=[AccuracyScorer(), GroundednessScorer()],
)

# Run the evaluation
import asyncio
results = asyncio.run(evaluation.evaluate(rag_model))

# Output in Weave dashboard:
# AccuracyScorer/score:     NaN   ← should be ~0.74
# GroundednessScorer/score: NaN   ← should be ~0.81
# Both columns NaN. Every run. No errors in logs.

How a scalar return value causes NaN aggregation

In weave 0.50.8, the Scorer base class defines a contract: the score() method must return a dict whose keys are the metric names to be aggregated. The canonical return shape is {"score": float_value}. The aggregation step collects the score() return value from every row in the evaluation dataset and computes the mean over the "score" key across all rows.

When score() returns a plain Python float instead of a dict, weave 0.50.8 does not raise a TypeError. It silently normalizes the scalar by wrapping it in a dict: {"output": 0.82}. The key used for the wrap is "output" — the same key weave uses internally for the model’s output value. The aggregation step then iterates over all per-row result dicts and attempts to compute the mean for the key "score". Because every row’s result dict has the key "output" but not "score", the aggregation finds zero matching values for "score". The mean of an empty series is NaN.

# Diagnosing the NaN: inspect per-row score() return values
# weave 0.50.8, Python 3.11

import weave
from weave import Scorer

weave.init("clearbrook-rag-eval-debug")

class AccuracyScorer(Scorer):
    @weave.op()
    def score(self, output: str, target: str) -> float:
        result = 1.0 if output.strip().lower() == target.strip().lower() else 0.0
        print(f"AccuracyScorer.score() returned: {result!r} (type: {type(result).__name__})")
        # Output: AccuracyScorer.score() returned: 1.0 (type: float)
        return result

# After running evaluation.evaluate() on a 5-row synthetic dataset:
# - Weave console shows 5 score() calls, all returning float.
# - weave internally wraps each float as {'output': float}.
# - Aggregation log (if verbose): computing mean over key 'score'; 0 values found; result: NaN.
# - Dashboard: AccuracyScorer/score = NaN.

# ─── Confirming the wrapping behavior via weave source inspection ───────────
# weave/flow/scorer.py (weave 0.50.8):
#
# def _scorer_call_result_to_dict(result: Any) -> dict:
#     if isinstance(result, dict):
#         return result
#     # scalar return: wrap in {'output': value}
#     return {"output": result}
#
# The aggregation step:
# def _aggregate_scorer_results(results: list[dict], scorer_name: str) -> dict:
#     # collects all values under key 'score' from each result dict
#     scores = [r.get("score") for r in results if r.get("score") is not None]
#     if not scores:
#         return {"mean": float("nan")}  # ← produces the NaN shown in dashboard
#     return {"mean": sum(scores) / len(scores)}
#
# Root cause confirmed: wrapping scalar as {'output': val} means
# r.get("score") returns None for every row → scores list empty → NaN.

Why the Weave dashboard shows NaN with no error

The four-week delay in diagnosis happened for a compound reason. First, weave 0.50.8 does not raise an error when score() returns a scalar — the evaluation run completes with a success status. Second, the Weave dashboard renders NaN in a table cell using the same font and layout as a numeric value; there is no red border, warning icon, or tooltip. Third, the team’s first hypothesis was a dashboard rendering bug in the Weave UI, which they filed as a support ticket. The support response pointed to a known issue with decimal precision rendering, which was unrelated. Fourth, since both scorers returned NaN simultaneously, the team did not isolate one scorer as the source — they assumed the problem was in the evaluation run itself.

The retainer developer confirmed the root cause in 30 minutes by adding a print statement to AccuracyScorer.score(), observing that the method returned a float, and then reading the weave/flow/scorer.py source to confirm the wrapping behavior.

The fix: return a dict with key “score” from score()

# eval_pipeline.py — FIXED
# Clearbrook ML Ops Ltd.
# weave 0.50.8, Python 3.11

import weave
from weave import Scorer

weave.init("clearbrook-rag-eval")

class AccuracyScorer(Scorer):
    """Score LLM answer accuracy against ground-truth target."""

    @weave.op()
    def score(self, output: str, target: str) -> dict:
        # ← FIX: return dict with key 'score', not a plain float.
        # weave 0.50.8 aggregation step looks for key 'score' across all rows.
        # Any additional keys in the dict are also collected and aggregated:
        #   {'score': 0.5, 'exact_match': 0.0} → two columns in the dashboard.
        predicted = output.strip().lower()
        expected = target.strip().lower()
        if predicted == expected:
            accuracy = 1.0
            exact = 1.0
        elif expected in predicted or predicted in expected:
            accuracy = 0.5
            exact = 0.0
        else:
            accuracy = 0.0
            exact = 0.0
        return {
            "score": accuracy,       # primary metric: shown in dashboard as mean
            "exact_match": exact,    # secondary metric: also aggregated
        }

class GroundednessScorer(Scorer):
    """Score whether the answer is grounded in the retrieved context."""

    @weave.op()
    def score(self, output: str, context: str) -> dict:
        # ← FIX: return {'score': float} instead of bare float.
        output_words = set(output.lower().split())
        context_words = set(context.lower().split())
        overlap = len(output_words & context_words)
        if not output_words:
            groundedness = 0.0
        else:
            groundedness = min(1.0, overlap / len(output_words))
        return {"score": groundedness}

evaluation = weave.Evaluation(
    dataset=eval_dataset,
    scorers=[AccuracyScorer(), GroundednessScorer()],
)

import asyncio
results = asyncio.run(evaluation.evaluate(rag_model))

# Output in Weave dashboard after fix:
# AccuracyScorer/score:        0.742   ← numeric, not NaN
# AccuracyScorer/exact_match:  0.631   ← bonus column from extra dict key
# GroundednessScorer/score:    0.814   ← numeric, not NaN

Extending with a custom summarize() for non-mean aggregation

For cases where the mean is the wrong aggregation (e.g., F1 score requires precision and recall to be aggregated separately before computing F1), weave.Scorer supports a summarize() classmethod. This method receives the list of all per-row score() return dicts and returns a single summary dict that overrides the default mean aggregation.

# Custom summarize() for F1 score aggregation
# weave 0.50.8, Python 3.11

import weave
from weave import Scorer

class F1Scorer(Scorer):
    """Compute token-level F1 between output and target."""

    @weave.op()
    def score(self, output: str, target: str) -> dict:
        output_tokens = set(output.lower().split())
        target_tokens = set(target.lower().split())
        if not output_tokens or not target_tokens:
            return {"score": 0.0, "precision": 0.0, "recall": 0.0}
        tp = len(output_tokens & target_tokens)
        precision = tp / len(output_tokens)
        recall = tp / len(target_tokens)
        f1 = (2 * precision * recall / (precision + recall)) if (precision + recall) > 0 else 0.0
        return {
            "score": f1,              # default aggregation: mean F1 across rows
            "precision": precision,   # also tracked per-row
            "recall": recall,
        }

    @classmethod
    def summarize(cls, score_rows: list[dict]) -> dict:
        # Override default mean aggregation with macro-averaged F1.
        # score_rows: list of per-row score() return dicts.
        # Each dict has keys: score, precision, recall.
        all_precision = [r["precision"] for r in score_rows if "precision" in r]
        all_recall = [r["recall"] for r in score_rows if "recall" in r]
        if not all_precision:
            return {"f1": float("nan"), "precision": float("nan"), "recall": float("nan")}
        mean_p = sum(all_precision) / len(all_precision)
        mean_r = sum(all_recall) / len(all_recall)
        macro_f1 = (2 * mean_p * mean_r / (mean_p + mean_r)) if (mean_p + mean_r) > 0 else 0.0
        return {
            "f1": macro_f1,           # shown in dashboard as F1Scorer/f1
            "precision": mean_p,
            "recall": mean_r,
        }

# With summarize() defined:
# Dashboard shows F1Scorer/f1, F1Scorer/precision, F1Scorer/recall
# Not the default mean of 'score' — the summarize() output overrides it.

Validation

# Smoke test: run eval on 5-row synthetic dataset to confirm non-NaN scores
# weave 0.50.8, Python 3.11

import weave
import asyncio
from weave import Scorer

weave.init("clearbrook-rag-eval-smoke")

class AccuracyScorer(Scorer):
    @weave.op()
    def score(self, output: str, target: str) -> dict:
        match = output.strip().lower() == target.strip().lower()
        return {"score": 1.0 if match else 0.0}

# 5-row synthetic dataset
smoke_dataset = [
    {"question": "What is 2+2?", "context": "Basic arithmetic.", "target": "4"},
    {"question": "Capital of France?", "context": "European capitals.", "target": "Paris"},
    {"question": "Color of sky?", "context": "Atmosphere physics.", "target": "blue"},
    {"question": "H2O formula?", "context": "Chemistry basics.", "target": "water"},
    {"question": "Largest planet?", "context": "Solar system.", "target": "Jupiter"},
]

class MockRAGModel(weave.Model):
    @weave.op()
    def predict(self, question: str, context: str) -> str:
        # Mock: returns the target for half the questions
        return {"What is 2+2?": "4", "Capital of France?": "Paris"}.get(question, "unknown")

evaluation = weave.Evaluation(
    dataset=smoke_dataset,
    scorers=[AccuracyScorer()],
)
results = asyncio.run(evaluation.evaluate(MockRAGModel()))

mean_score = results["AccuracyScorer"]["score"]["mean"]
assert not (mean_score != mean_score), f"AccuracyScorer mean is NaN — check score() return type"
# mean_score = 0.4 (2 correct out of 5)
# Assertion passes. Smoke test confirms non-NaN scores.
# Safe to run on full 600-document dataset.

# Full eval results after fix (October 29 2026):
# AccuracyScorer/score:        0.742   ← matches domain-expert manual annotation baseline 0.74
# AccuracyScorer/exact_match:  0.631
# GroundednessScorer/score:    0.814
# 4 weeks of NaN resolved in 2.0h retainer engagement.

Related retainer posts

• LlamaIndex developer on retainer — custom node post-processor mutates shared node metadata dict in-place — subsequent retrieval queries return stale filtered results — llamaindex-core 0.10.43
• DSPy developer on retainer — compiled module teleprompter overwrites demonstration cache on every call — optimized few-shot examples lost between invocations — dspy-ai 2.4.14
• Ragas developer on retainer — context precision metric always returns 1.0 when ground truth contexts and retrieved contexts share no overlap — ragas 0.1.9 scoring logic inverted