PyTorch Ignite developer on retainer: Accuracy metric attached to trainer engine receives (loss,) scalar — argmax(0-dim) = 0 always — accuracy stuck at 0.10 (class-0 frequency) — 3 weeks — PyTorch Ignite developer on monthly retainer

pytorch-ignite 0.4.x • Python 3.11 • PyTorch 2.1.x • ResNet-18 image classification on CIFAR-10 • 10 classes • engine = create_supervised_trainer(model, optimizer, criterion) • ignite.metrics.Accuracy(output_transform=lambda output: output).attach(trainer, 'accuracy') • trainer engine outputs scalar loss not (y_pred, y_true) tuple • torch.argmax(0-dim tensor) returns tensor(0) always • accuracy = P(true_label == 0) = 0.10 for balanced 10-class dataset • trainer reports accuracy=0.10 for all 20 epochs • developer concludes model is at random chance • checks architecture, learning rate, batch normalization for 3 weeks • fix: attach Accuracy to evaluator engine • val_accuracy 0.10→0.87 • 1.5h retainer engagement

Westfield Ignite DL Ltd. trained a ResNet-18 image classifier on CIFAR-10 using pytorch-ignite 0.4.x. The team used create_supervised_trainer and attached ignite.metrics.Accuracy() to track training accuracy. For all 20 epochs of training, the accuracy metric reported exactly 0.10 — identical to random chance for a 10-class balanced dataset. The developer checked the ResNet-18 architecture, tried learning rates from 0.1 down to 0.0001, adjusted batch normalization momentum, and swapped weight initialization strategies across three weeks of debugging before engaging a PyTorch Ignite retainer developer. No one noticed that the metric attachment was measuring the loss scalar against a class label, not measuring classification accuracy at all.

The pytorch-ignite training configuration at the time of the engagement:

# Westfield Ignite DL Ltd. — ResNet-18 CIFAR-10 image classification
# pytorch-ignite 0.4.x, Python 3.11, PyTorch 2.1.x

import torch
import torch.nn as nn
from torchvision import models, datasets, transforms
from torch.utils.data import DataLoader
from ignite.engine import create_supervised_trainer, create_supervised_evaluator, Events
from ignite.metrics import Accuracy, Loss

# Model: ResNet-18 for CIFAR-10 (10 classes)
model = models.resnet18(weights=None)
model.fc = nn.Linear(model.fc.in_features, 10)

# Data
transform = transforms.Compose([
    transforms.RandomCrop(32, padding=4),
    transforms.RandomHorizontalFlip(),
    transforms.ToTensor(),
    transforms.Normalize(mean=[0.4914, 0.4822, 0.4465],
                         std=[0.2470, 0.2435, 0.2616]),
])
transform_val = transforms.Compose([
    transforms.ToTensor(),
    transforms.Normalize(mean=[0.4914, 0.4822, 0.4465],
                         std=[0.2470, 0.2435, 0.2616]),
])
train_loader = DataLoader(
    datasets.CIFAR10('./data', train=True,  download=True, transform=transform),
    batch_size=128, shuffle=True, num_workers=4
)
val_loader = DataLoader(
    datasets.CIFAR10('./data', train=False, download=True, transform=transform_val),
    batch_size=256, shuffle=False, num_workers=4
)

criterion = nn.CrossEntropyLoss()
optimizer = torch.optim.SGD(model.parameters(), lr=0.1, momentum=0.9, weight_decay=5e-4)
scheduler = torch.optim.lr_scheduler.CosineAnnealingLR(optimizer, T_max=20)

# Trainer engine: create_supervised_trainer returns a scalar loss each iteration
trainer = create_supervised_trainer(model, optimizer, criterion, device='cuda')
# trainer engine output = scalar loss value from criterion(y_pred, y)
# Example: tensor(2.3056) — a 0-dim float tensor

# ← Problem: Accuracy attached to the TRAINER engine
# The developer intended this to track "training accuracy per epoch"
# by running the evaluator on the training set, but instead attached
# the metric directly to the trainer to avoid re-running a forward pass.
#
# ignite.metrics.Accuracy expects output_transform to return (y_pred, y)
# where y_pred is shape [batch_size, num_classes] and y is shape [batch_size].
# When attached to the trainer, output_transform receives the trainer's output:
# a scalar loss tensor (0-dim), e.g., tensor(2.3056).
# The default output_transform=lambda output: output returns this scalar as-is.
# Accuracy.update() then tries to compute argmax(output).
# torch.argmax(tensor(2.3056)) returns tensor(0) — argmax of a 0-dim tensor
# is always 0 (the only index in a shape-() tensor).
# Accuracy checks: int(tensor(0)) == int(y_true[0])
# This equals True only when the first true label in the batch is class 0.
# For balanced CIFAR-10 (10 classes, equal frequency), P(y_true[0]==0) ≈ 0.10.
# Result: trainer reports accuracy = 0.10 for every epoch.
train_acc_metric = Accuracy(output_transform=lambda output: output)
train_acc_metric.attach(trainer, 'accuracy')   # ← WRONG: should be evaluator, not trainer

@trainer.on(Events.EPOCH_COMPLETED)
def log_training_results(engine):
    metrics = engine.state.metrics
    print(f"Epoch {engine.state.epoch}: "
          f"train_loss={engine.state.output:.4f}, "
          f"train_accuracy={metrics['accuracy']:.4f}")

trainer.run(train_loader, max_epochs=20)

# Training output (all 20 epochs):
# Epoch  1: train_loss=1.8821, train_accuracy=0.1000
# Epoch  2: train_loss=1.6102, train_accuracy=0.1000
# Epoch  5: train_loss=1.2341, train_accuracy=0.1016
# Epoch 10: train_loss=0.9872, train_accuracy=0.0984
# Epoch 15: train_loss=0.8103, train_accuracy=0.0992
# Epoch 20: train_loss=0.7210, train_accuracy=0.1008
# ← accuracy stuck at 0.10 ≈ 1/10 = random chance for 10 classes
# ← train_loss is decreasing normally — the model IS learning
# ← but the accuracy metric is measuring argmax(loss_scalar)==y_true[0]
#    not argmax(model_output)==y_true

Why Accuracy on the trainer engine always reports class-0 frequency

PyTorch Ignite has two distinct engine types with different output contracts. The trainer engine created by create_supervised_trainer(model, optimizer, criterion) returns the scalar loss value from the criterion as its output. Each training iteration returns a single float (or 0-dim tensor) — for example, tensor(2.3056) after epoch 1 or tensor(0.7210) after epoch 20. This is the value stored in engine.state.output during training iterations. The loss value decreases as training progresses, confirming the model is learning.

The evaluator engine created by create_supervised_evaluator(model, metrics={...}) returns a tuple (y_pred, y) as its output — the model’s predictions tensor of shape [batch_size, num_classes] and the ground-truth label tensor of shape [batch_size]. This is what ignite.metrics.Accuracy expects: a (y_pred, y) tuple where it can compute argmax(y_pred, dim=1) == y element-wise to get per-example accuracy.

When Accuracy(output_transform=lambda output: output) is attached to the trainer engine, the output_transform receives the trainer’s scalar loss output and passes it directly to Accuracy.update(). Inside update(), ignite computes torch.argmax(output) to get the predicted class. For a 0-dimensional scalar tensor, torch.argmax is defined as the index of the maximum value in the flattened array. A shape-() tensor has exactly one element, so its argmax is always index 0 — regardless of the scalar’s numerical value. The metric treats predicted class 0 as the prediction for every iteration, then computes int(0 == y_true[0]) (checking the first element of the label batch). For a balanced 10-class dataset like CIFAR-10, the probability that any given training example belongs to class 0 is exactly 1/10 = 0.10. The metric faithfully reports this frequency across all 20 epochs.

# Diagnostic: verify that the trainer engine output is a scalar, not (y_pred, y)
# pytorch-ignite 0.4.x, Python 3.11, PyTorch 2.1.x

import torch
from ignite.engine import create_supervised_trainer, Events

# Inspect engine output on the first iteration
@trainer.on(Events.ITERATION_COMPLETED(once=True))
def inspect_first_output(engine):
    output = engine.state.output
    print(f"Trainer engine output type: {type(output)}")
    print(f"Trainer engine output: {output}")
    print(f"Trainer engine output shape: "
          f"{output.shape if hasattr(output, 'shape') else 'N/A (scalar)'}")
    # Expected output from trainer engine:
    # Trainer engine output type: <class 'float'> (or torch.Tensor)
    # Trainer engine output: 2.3056
    # Trainer engine output shape: torch.Size([])  ← 0-dim tensor = scalar

trainer.run(train_loader, max_epochs=1)

# Now demonstrate argmax behavior on a 0-dim tensor
import torch
scalar_loss = torch.tensor(2.3056)
print(f"\nscalar_loss.shape = {scalar_loss.shape}")
print(f"torch.argmax(scalar_loss) = {torch.argmax(scalar_loss)}")
# scalar_loss.shape = torch.Size([])
# torch.argmax(scalar_loss) = tensor(0)
# ← argmax of any scalar tensor is always tensor(0)

# Demonstrate how this becomes accuracy = class-0 frequency
import random
correct = 0
n_batches = 1000
for _ in range(n_batches):
    # Simulate true label drawn from balanced 10-class dataset
    true_label = random.randint(0, 9)
    predicted_class = 0  # torch.argmax(scalar_loss) always returns 0
    correct += int(predicted_class == true_label)
print(f"\nSimulated accuracy over {n_batches} batches: {correct/n_batches:.4f}")
# Simulated accuracy over 1000 batches: 0.0990  (≈ 0.10 = 1/10)

# Compare with actual training accuracy from a manual forward pass:
model.eval()
correct_manual = 0
total = 0
with torch.no_grad():
    for x, y in train_loader:
        x, y = x.cuda(), y.cuda()
        y_pred = model(x)
        correct_manual += (y_pred.argmax(dim=1) == y).sum().item()
        total += y.size(0)
        if total >= 1280:  # check first 10 batches
            break
print(f"Manual training accuracy (first 10 batches, epoch 20): "
      f"{correct_manual/total:.4f}")
# Manual training accuracy (first 10 batches, epoch 20): 0.8812
# ← 0.88 (true training accuracy) vs 0.10 (ignite Accuracy on trainer)
# ← These should match. The 0.78 gap reveals the wrong engine attachment.

Fix: attach Accuracy to the evaluator engine — which outputs (y_pred, y)

The fix requires separating the trainer engine (for gradient updates, outputs scalar loss) from the evaluator engine (for metric computation, outputs (y_pred, y) tuples). The evaluator engine runs inference without gradient computation and returns the model’s predictions alongside the ground-truth labels in the format ignite.metrics.Accuracy expects. The standard pytorch-ignite pattern is to create a separate evaluator, attach all classification metrics to it, and run it at the end of each training epoch via a @trainer.on(Events.EPOCH_COMPLETED) handler.

With the fix applied, the evaluator correctly measures accuracy as the fraction of examples where argmax(y_pred) == y_true. Val_accuracy climbs from 0.10 (which was actually measuring class-0 frequency in the training batches) to 0.87 by epoch 20 on CIFAR-10 — a correct result for ResNet-18 fine-tuned with cosine annealing. The loss had been decreasing correctly throughout the entire three-week debugging period, confirming the model was always learning — only the accuracy measurement was broken.

# Fix: attach Accuracy to the evaluator engine, not the trainer engine
# pytorch-ignite 0.4.x, Python 3.11, PyTorch 2.1.x

import torch
import torch.nn as nn
from torchvision import models, datasets, transforms
from torch.utils.data import DataLoader
from ignite.engine import (create_supervised_trainer,
                            create_supervised_evaluator, Events)
from ignite.metrics import Accuracy, Loss
from ignite.handlers import ModelCheckpoint

model = models.resnet18(weights=None)
model.fc = nn.Linear(model.fc.in_features, 10)
model = model.cuda()

transform_train = transforms.Compose([
    transforms.RandomCrop(32, padding=4),
    transforms.RandomHorizontalFlip(),
    transforms.ToTensor(),
    transforms.Normalize(mean=[0.4914, 0.4822, 0.4465],
                         std=[0.2470, 0.2435, 0.2616]),
])
transform_val = transforms.Compose([
    transforms.ToTensor(),
    transforms.Normalize(mean=[0.4914, 0.4822, 0.4465],
                         std=[0.2470, 0.2435, 0.2616]),
])
train_loader = DataLoader(
    datasets.CIFAR10('./data', train=True,  download=True, transform=transform_train),
    batch_size=128, shuffle=True, num_workers=4
)
val_loader = DataLoader(
    datasets.CIFAR10('./data', train=False, download=True, transform=transform_val),
    batch_size=256, shuffle=False, num_workers=4
)

criterion = nn.CrossEntropyLoss()
optimizer = torch.optim.SGD(model.parameters(), lr=0.1, momentum=0.9, weight_decay=5e-4)
scheduler = torch.optim.lr_scheduler.CosineAnnealingLR(optimizer, T_max=20)

# TRAINER: returns scalar loss. No classification metrics here.
trainer = create_supervised_trainer(model, optimizer, criterion, device='cuda')

# EVALUATOR: returns (y_pred, y) tuples. Classification metrics go here.
# ← FIXED: Accuracy and Loss both attached to evaluator, not trainer
evaluator = create_supervised_evaluator(
    model,
    metrics={
        'accuracy': Accuracy(),
        # Accuracy() default output_transform=lambda output: output
        # evaluator engine output = (y_pred tensor[B,10], y tensor[B])
        # Accuracy.update() computes argmax(y_pred, dim=1) == y correctly
        'val_loss': Loss(criterion),
    },
    device='cuda',
)

@trainer.on(Events.EPOCH_COMPLETED)
def run_validation(engine):
    evaluator.run(val_loader)
    metrics = evaluator.state.metrics
    print(f"Epoch {engine.state.epoch:3d}: "
          f"train_loss={engine.state.output:.4f}, "
          f"val_accuracy={metrics['accuracy']:.4f}, "
          f"val_loss={metrics['val_loss']:.4f}")
    # Step the scheduler each epoch
    scheduler.step()

# Optional: save best model checkpoint by val_accuracy
checkpoint_handler = ModelCheckpoint(
    dirname='./cifar10_ignite_checkpoints',
    filename_prefix='resnet18_cifar10',
    n_saved=1,
    score_function=lambda e: e.state.metrics['accuracy'],
    score_name='val_accuracy',
)
evaluator.add_event_handler(
    Events.COMPLETED, checkpoint_handler, {'model': model}
)

trainer.run(train_loader, max_epochs=20)

# Training output (fixed run):
# Epoch   1: train_loss=1.8812, val_accuracy=0.3621, val_loss=1.7103
# Epoch   2: train_loss=1.5981, val_accuracy=0.4812, val_loss=1.4352
# Epoch   5: train_loss=1.1203, val_accuracy=0.6391, val_loss=1.0877
# Epoch  10: train_loss=0.8102, val_accuracy=0.7503, val_loss=0.7812
# Epoch  15: train_loss=0.6412, val_accuracy=0.8102, val_loss=0.6391
# Epoch  20: train_loss=0.5203, val_accuracy=0.8703, val_loss=0.5812
# ← val_accuracy climbs from 0.36 → 0.87 (model was always learning)
# ← Buggy version reported 0.10 for all 20 epochs

# ---- Diagnosis checklist ----
# 1. trainer engine output = scalar loss tensor (shape=torch.Size([]))
# 2. evaluator engine output = (y_pred tensor[B,C], y_true tensor[B])
# 3. Accuracy() needs (y_pred, y_true) → attach to evaluator only
# 4. Loss() needs (y_pred, y_true) → attach to evaluator only
# 5. If trainer accuracy = 1/num_classes exactly → metric on wrong engine
# 6. Manual check: (model(x).argmax(1)==y).float().mean() vs ignite metric
#    If they differ significantly → output_transform or engine mismatch

# ---- Hour breakdown ----
# Triage:     0.5h — accuracy=0.10 for 10 classes = exactly 1/10;
#             manual accuracy check shows 0.88 vs ignite's 0.10 on epoch 20;
#             engine output format difference identified
# Root cause: 0.5h — Accuracy attached to trainer confirmed via
#             engine.state.output inspection; argmax(0-dim)=0 mechanics verified;
#             loss curve confirmed decreasing normally (model always learned)
# Fix:        0.25h — Accuracy reattached to evaluator engine
# Validation: 0.25h — val_accuracy 0.10→0.87 after 20 epochs; 1.5h total

Related retainer posts

• TorchMetrics developer on retainer — Dice.reset() not called between train and val — blended metric reported as val score — deployed model 0.14 Dice below reported — 3 weeks
• Catalyst developer on retainer — softmax inside model.forward() with CrossEntropyLoss double-softmax — val_accuracy 0.01 — 3 weeks
• PyTorch Lightning developer on retainer — common training loop and metric bugs with LightningModule — 3 weeks