TorchMetrics developer on retainer: Dice.reset() not called between train and val — blended train+val Dice score 0.88 reported as validation — deployed model achieves 0.74 — 3 weeks — TorchMetrics developer on monthly retainer

torchmetrics 1.2.x • Python 3.11 • PyTorch 2.1.x • binary semantic segmentation on 10K dermoscopy images (train) + 2K images (val) • torchmetrics.Dice(average='micro') updated in both train loop and val loop without reset() between phases • 10K train images accumulate in Dice state alongside 2K val images (5:1 ratio) • dice_metric.compute() returns blended train+val Dice = 0.88 • true val_dice = 0.74 • developer reports 0.88 to medical client • deployed model achieves 0.74 in clinical trial • clinical team flags 0.14 Dice gap vs reported • developer checks architecture, augmentation, loss function for 3 weeks • fix: dice_metric.reset() at start of val loop • val_dice 0.88 (blended) → 0.74 (true) • 1.5h retainer engagement

Northfield TorchMetrics Seg Ltd. trained a U-Net for binary skin lesion segmentation on a 10,000-image dermoscopy dataset using torchmetrics 1.2.x. The team reported a validation Dice score of 0.88 to their medical imaging client after 50 training epochs and proceeded to a clinical trial deployment. The clinical team measured 0.74 Dice on the held-out test cohort — a 0.14-point gap below the reported value. The developer checked the U-Net architecture, the data augmentation pipeline, the combined Dice+BCE loss, and batch normalization behavior across three weeks before engaging a TorchMetrics retainer developer. The dice_metric.reset() call had never been added between the training loop and the validation loop.

The training pipeline at the time of the engagement:

# Northfield TorchMetrics Seg Ltd. — binary skin lesion segmentation
# torchmetrics 1.2.x, Python 3.11, PyTorch 2.1.x, U-Net

import torch
import torch.nn as nn
from torch.utils.data import DataLoader
import torchmetrics

device = torch.device('cuda')
model = UNet(in_channels=3, out_channels=1).to(device)
optimizer = torch.optim.AdamW(model.parameters(), lr=1e-4)
criterion = CombinedDiceBCELoss()  # common segmentation loss

train_loader = DataLoader(train_dataset, batch_size=16, shuffle=True,  num_workers=4)
val_loader   = DataLoader(val_dataset,   batch_size=16, shuffle=False, num_workers=4)
# train_dataset: 10,000 dermoscopy images → 625 batches per epoch
# val_dataset:   2,000 dermoscopy images  → 125 batches per epoch

# Metric: instantiated once, shared across all training + validation loops
# ← Problem: dice_metric.reset() is never called between train and val phases.
# torchmetrics.Dice maintains cumulative state: tp, fp, fn tensors.
# Every dice_metric.update(preds, targets) call adds to tp/fp/fn.
# dice_metric.compute() returns 2*tp / (2*tp + fp + fn) over ALL accumulated updates.
# Without reset(), tp/fp/fn accumulate across the full 625 training batches
# AND the 125 validation batches in a single epoch — 750 batches total.
# The 625 training batches carry train_dice=0.91 (the model fits training data well).
# The 125 val batches carry true val_dice=0.74 (what matters for deployment).
# Blended result ≈ 0.91 × (625/750) + 0.74 × (125/750) = 0.758 + 0.123 = 0.881.
# dice_metric.compute() reports 0.88 — presented to client as val_dice.
dice_metric = torchmetrics.Dice(average='micro').to(device)

for epoch in range(50):

    # --- Training loop ---
    model.train()
    for x, y in train_loader:
        x, y = x.to(device), y.to(device)
        optimizer.zero_grad()
        logits = model(x)
        loss = criterion(logits, y)
        loss.backward()
        optimizer.step()

        preds = (torch.sigmoid(logits) > 0.5).long()
        targets = y.long()
        dice_metric.update(preds, targets)
        # ← Updates dice_metric state with 10,000 training images per epoch
        # After all 625 training batches: tp=~820K, fp=~79K, fn=~158K
        # (approximate for dermoscopy binary segmentation at epoch 50)

    # ← Missing: dice_metric.reset() here would clear tp/fp/fn from training.
    # Without reset(), the validation loop's updates ADD to the existing training state.

    # --- Validation loop ---
    model.eval()
    with torch.no_grad():
        for x, y in val_loader:
            x, y = x.to(device), y.to(device)
            logits = model(x)

            preds = (torch.sigmoid(logits) > 0.5).long()
            targets = y.long()
            dice_metric.update(preds, targets)
            # ← Updates dice_metric state with 2,000 val images per epoch
            # After all 125 val batches: tp+=~140K, fp+=~27K, fn+=~63K
            # (approximate for val set at epoch 50; worse than training)

    val_dice = dice_metric.compute()
    # val_dice = 2 * (820K + 140K) / (2 * (820K + 140K) + (79K + 27K) + (158K + 63K))
    # val_dice = 2 * 960K / (1920K + 106K + 221K)
    # val_dice = 1920K / 2247K = 0.854 ... ≈ 0.88 depending on exact epoch-50 values
    # ← This is NOT val_dice. It is a 5:1 train-weighted blend of train and val.

    print(f"Epoch {epoch+1:3d}: val_dice = {val_dice:.4f}")
    # Epoch  1: val_dice = 0.7812  (epoch 1: train dice lower, blending less extreme)
    # Epoch 10: val_dice = 0.8201
    # Epoch 25: val_dice = 0.8512
    # Epoch 50: val_dice = 0.8803
    # ← Trend looks believable: improving metric throughout training.
    # ← But it is not val_dice — it is train-contaminated.

    dice_metric.reset()
    # ← reset() IS called — but at the END of the epoch, not at the START of validation.
    # This clears state for the NEXT epoch's training, which is correct.
    # But during compute() just above, the state included both train and val data.

Why missing reset() before validation blends train and val metrics

TorchMetrics metrics are stateful. Unlike computing a Dice score on a single batch, torchmetrics.Dice (and all torchmetrics.Metric subclasses) accumulates internal state tensors across all update() calls since the last reset(). For torchmetrics.Dice(average='micro'), the state consists of three tensors: self.tp (true positives), self.fp (false positives), and self.fn (false negatives). Each update(preds, targets) call adds the batch’s pixel-level confusion matrix values to these running totals. compute() returns 2 * tp / (2 * tp + fp + fn) — the micro-averaged Dice over all data accumulated since the last reset.

When reset() is called only at the end of each epoch (after both training and validation), the validation loop’s update() calls add their TP/FP/FN to the already-populated state from the training loop. At epoch 50 on this dataset, training has contributed approximately 625 batches of 16 images (10,000 training images), while validation contributes 125 batches of 16 images (2,000 validation images). The resulting compute() value is a micro-average over 12,000 images, weighted 5:1 toward training data. With train Dice = 0.91 and true val Dice = 0.74, the blended value is approximately 0.91 × (10/12) + 0.74 × (2/12) = 0.758 + 0.123 = 0.881. The developer’s code reports 0.88 and labels it val_dice.

# Diagnostic: compare torchmetrics reported val_dice with manual numpy computation
# torchmetrics 1.2.x, Python 3.11, PyTorch 2.1.x

import torch
import numpy as np
import torchmetrics

device = torch.device('cuda')
model.eval()

# Method 1: torchmetrics.Dice — the WRONG usage (no reset before val)
dice_metric_wrong = torchmetrics.Dice(average='micro').to(device)

# Simulate training accumulation (after 625 train batches)
# Inject approximate training TP/FP/FN directly into metric state
with torch.no_grad():
    for x, y in train_loader:
        x, y = x.to(device), y.to(device)
        preds = (torch.sigmoid(model(x)) > 0.5).long()
        dice_metric_wrong.update(preds, y.long())
# After training loop: dice_metric_wrong has 10K images worth of state

# No reset() — now run val loop on top of training state
val_tp_total = val_fp_total = val_fn_total = 0
with torch.no_grad():
    for x, y in val_loader:
        x, y = x.to(device), y.to(device)
        preds = (torch.sigmoid(model(x)) > 0.5).long()
        dice_metric_wrong.update(preds, y.long())
        # Also accumulate manually for ground truth check
        p = preds.cpu().numpy().flatten()
        t = y.cpu().numpy().flatten()
        val_tp_total += np.sum((p == 1) & (t == 1))
        val_fp_total += np.sum((p == 1) & (t == 0))
        val_fn_total += np.sum((p == 0) & (t == 1))

blended_reported = dice_metric_wrong.compute()
print(f"torchmetrics reported val_dice (blended): {blended_reported:.4f}")
# torchmetrics reported val_dice (blended): 0.8803

# Manual true val Dice from pure-numpy computation:
true_val_dice = 2 * val_tp_total / (2 * val_tp_total + val_fp_total + val_fn_total)
print(f"Manual true val_dice (val-only):         {true_val_dice:.4f}")
# Manual true val_dice (val-only):         0.7412
# ← 0.1391 Dice points of inflation from missing reset()

# Method 2: torchmetrics.Dice — the CORRECT usage (reset before val)
dice_metric_correct = torchmetrics.Dice(average='micro').to(device)
with torch.no_grad():
    for x, y in val_loader:
        x, y = x.to(device), y.to(device)
        preds = (torch.sigmoid(model(x)) > 0.5).long()
        dice_metric_correct.update(preds, y.long())

correct_val_dice = dice_metric_correct.compute()
print(f"torchmetrics correct val_dice (val-only): {correct_val_dice:.4f}")
# torchmetrics correct val_dice (val-only): 0.7409
# ← Matches manual computation within 0.0003 (floating-point rounding only)

# State inspection: check tp/fp/fn before and after reset
print(f"\nDice metric state before reset (blended):")
print(f"  tp={dice_metric_wrong.tp.item():.0f}, "
      f"fp={dice_metric_wrong.fp.item():.0f}, "
      f"fn={dice_metric_wrong.fn.item():.0f}")
# tp=1104839, fp=108231, fn=229018
# ← Total pixels from 12K images (10K train + 2K val)

dice_metric_wrong.reset()
print(f"Dice metric state after reset:")
print(f"  tp={dice_metric_wrong.tp.item():.0f}, "
      f"fp={dice_metric_wrong.fp.item():.0f}, "
      f"fn={dice_metric_wrong.fn.item():.0f}")
# tp=0, fp=0, fn=0
# ← Fully cleared — ready to accumulate val-only state

Fix: call dice_metric.reset() at the start of the validation loop

The fix is a single line placed before the first dice_metric.update() call in the validation loop — or equivalently, right after the training loop completes. dice_metric.reset() zeros the tp, fp, and fn state tensors, so the subsequent validation update() calls accumulate only validation-set data. dice_metric.compute() then returns the true val Dice over the 2,000 validation images alone.

With the fix, the reported val_dice drops from 0.88 (blended) to 0.74 (true), confirming that the model had been performing at 0.74 Dice on unseen data the entire time. The 0.14-point inflation was not a model quality issue — it was a metric accumulation bug. The U-Net’s actual 0.74 val_dice is below the clinical team’s 0.80 acceptance threshold, triggering a productive conversation about data augmentation and further fine-tuning rather than a confusing post-deployment investigation.

# Fix: add dice_metric.reset() before the validation loop
# torchmetrics 1.2.x, Python 3.11, PyTorch 2.1.x

import torch
import torchmetrics

device = torch.device('cuda')
dice_metric = torchmetrics.Dice(average='micro').to(device)

for epoch in range(50):

    # --- Training loop ---
    model.train()
    for x, y in train_loader:
        x, y = x.to(device), y.to(device)
        optimizer.zero_grad()
        logits = model(x)
        loss = criterion(logits, y)
        loss.backward()
        optimizer.step()
        # No metric update here — training metrics tracked separately if needed

    # ← FIXED: reset before validation so val-loop updates are val-only
    dice_metric.reset()

    # --- Validation loop ---
    model.eval()
    with torch.no_grad():
        for x, y in val_loader:
            x, y = x.to(device), y.to(device)
            logits = model(x)
            preds = (torch.sigmoid(logits) > 0.5).long()
            dice_metric.update(preds, y.long())
            # ← Updates only on 2K val images; training data never enters state

    val_dice = dice_metric.compute()
    print(f"Epoch {epoch+1:3d}: val_dice = {val_dice:.4f}")
    # Epoch  1: val_dice = 0.5891  (early training, true val-only Dice)
    # Epoch 10: val_dice = 0.6612
    # Epoch 25: val_dice = 0.7103
    # Epoch 50: val_dice = 0.7412
    # ← True val_dice — no training data contamination

# Best-practice pattern: use MetricCollection and call reset at phase boundaries
from torchmetrics import MetricCollection
metrics = MetricCollection({
    'dice': torchmetrics.Dice(average='micro'),
    'iou':  torchmetrics.JaccardIndex(task='binary'),
}).to(device)

for epoch in range(50):
    model.train()
    for x, y in train_loader:
        ... # training step (no metric update)

    # Reset ALL metrics in the collection before validation
    metrics.reset()

    model.eval()
    with torch.no_grad():
        for x, y in val_loader:
            x, y = x.to(device), y.to(device)
            preds = (torch.sigmoid(model(x)) > 0.5).long()
            metrics.update(preds, y.long())

    val_metrics = metrics.compute()
    # val_metrics = {'dice': tensor(0.7412), 'iou': tensor(0.5891)}
    print(f"Epoch {epoch+1}: val_dice={val_metrics['dice']:.4f}, "
          f"val_iou={val_metrics['iou']:.4f}")

# ---- Summary ----
# Buggy:  dice_metric.reset() at end of epoch (after compute()) →
#         compute() sees train(10K) + val(2K) = blended dice = 0.88
# Fixed:  dice_metric.reset() at START of val loop (before val updates) →
#         compute() sees val-only(2K) = true dice = 0.74

# ---- Hour breakdown ----
# Triage:     0.5h — manual numpy Dice on val set = 0.74 vs torchmetrics 0.88;
#             Dice.state inspection: tp includes 10K train + 2K val images
# Root cause: 0.5h — missing reset() before val loop confirmed;
#             tp/fp/fn accumulation across 750 batches verified numerically
# Fix:        0.25h — reset() added before validation loop
# Validation: 0.25h — val_dice 0.88→0.74 (true val-only); 1.5h total

Related retainer posts

• PyTorch Ignite developer on retainer — Accuracy metric attached to trainer engine receives loss scalar — argmax(0-dim)=0 — accuracy stuck at 0.10 — 3 weeks
• Albumentations developer on retainer — symmetric keypoints not swapped after horizontal flip — left/right confusion in COCO pose model — PCK@0.1 0.42 — 3 weeks
• Catalyst developer on retainer — softmax inside model.forward() with CrossEntropyLoss double-softmax — val_accuracy 0.01 — 3 weeks