PyTorch Lightning developer on retainer: ReduceLROnPlateau monitors 'val_loss' but validation_step() logs 'val/loss' — exact key mismatch — scheduler never fires — learning rate never decreases — 3 weeks — PyTorch Lightning developer on monthly retainer

pytorch-lightning 2.2.x • Python 3.11 • PyTorch 2.3.x • ResNet-50 image classification • SGD optimizer • ReduceLROnPlateau(patience=5, factor=0.1) • configure_optimizers() sets monitor='val_loss' • validation_step() logs self.log('val/loss', loss) (TensorBoard grouping convention) • Lightning exact string match: 'val_loss' != 'val/loss' • scheduler monitor metric never found • ReduceLROnPlateau never receives monitored value • LR stays at 0.1 for all 50 epochs • developer checks patience, factor, optimizer type — 3 weeks • fix: MONITOR_KEY constant; align log key to monitor key • 1.5h retainer engagement

Westfield Model Training Ltd. was training a ResNet-50 image classifier using PyTorch Lightning with a ReduceLROnPlateau learning rate scheduler. After the first epoch, validation loss plateaued, and the developer expected the scheduler to begin reducing the learning rate after five epochs (the configured patience). But across 50 epochs of training, the learning rate never decreased from its initial value of 0.1. The developer spent three weeks adjusting the scheduler patience, factor, and optimizer type. They switched from SGD to Adam, reduced the initial learning rate, and consulted the PyTorch Lightning documentation repeatedly. They engaged a PyTorch Lightning retainer developer.

The LightningModule configuration at the time of the engagement:

# model.py — Westfield Model Training Ltd.
# pytorch-lightning 2.2.x, Python 3.11, PyTorch 2.3.x

import torch
import torch.nn as nn
import pytorch_lightning as pl
from torch.optim.lr_scheduler import ReduceLROnPlateau

class ImageClassifier(pl.LightningModule):
    def __init__(self, num_classes: int, lr: float = 0.1):
        super().__init__()
        self.save_hyperparameters()
        self.model = resnet50(weights=None, num_classes=num_classes)
        self.criterion = nn.CrossEntropyLoss()

    def training_step(self, batch, batch_idx):
        x, y = batch
        logits = self.model(x)
        loss = self.criterion(logits, y)
        # TensorBoard grouping convention: 'train/loss' groups under 'train'
        self.log('train/loss', loss, on_step=True, on_epoch=True, prog_bar=True)
        return loss

    def validation_step(self, batch, batch_idx):
        x, y = batch
        logits = self.model(x)
        loss = self.criterion(logits, y)
        acc = (logits.argmax(dim=1) == y).float().mean()
        # TensorBoard convention — logs under 'val/' group
        self.log('val/loss', loss, on_epoch=True, sync_dist=True)
        self.log('val/acc',  acc,  on_epoch=True, sync_dist=True, prog_bar=True)
        return loss

    def configure_optimizers(self):
        optimizer = torch.optim.SGD(
            self.parameters(),
            lr=self.hparams.lr,
            momentum=0.9,
            weight_decay=1e-4,
        )
        scheduler = ReduceLROnPlateau(
            optimizer,
            mode='min',
            factor=0.1,
            patience=5,
            min_lr=1e-6,
        )
        return {
            'optimizer': optimizer,
            'lr_scheduler': {
                'scheduler': scheduler,
                'monitor': 'val_loss',   # ← expects exact metric key 'val_loss'
                'interval': 'epoch',
                'frequency': 1,
            },
        }
        # Bug: validation_step() logs 'val/loss', not 'val_loss'
        # Lightning exact string match fails
        # ReduceLROnPlateau monitor metric never found
        # Scheduler never fires; LR stays at 0.1 forever

Why Lightning's scheduler monitoring requires an exact metric key match

PyTorch Lightning's metric-based scheduler support works through an exact string lookup in the Trainer's logged metrics dictionary. After each validation epoch, Lightning collects all metrics logged via self.log() into a flat dictionary keyed by the exact string passed to self.log(). When the scheduler's monitor key is configured, Lightning looks up that exact string in the metrics dictionary to get the current value for the scheduler step. There is no normalization of slashes, underscores, or namespaces — 'val_loss' and 'val/loss' are different dictionary keys.

In PyTorch Lightning 2.2.x, when the monitor key is not found in the logged metrics, the behavior is silent: the scheduler is not stepped, and no error or warning is emitted at the default logging level. A debug-level message is logged internally, but this requires setting the Lightning log level to DEBUG to see. The consequence is that ReduceLROnPlateau receives no monitored value and never reduces the learning rate. Training continues at the initial learning rate indefinitely.

# Diagnosing the ReduceLROnPlateau key mismatch
# pytorch-lightning 2.2.x

import pytorch_lightning as pl
import logging

# Step 1: enable Lightning debug logging to see scheduler warnings
logging.getLogger("pytorch_lightning").setLevel(logging.DEBUG)
# After running one validation epoch, look for:
# "Metric val_loss is not in logged metrics"
# This confirms the key mismatch.

# Step 2: inspect logged metrics at end of validation epoch
# Use an on_validation_epoch_end hook to print all logged metrics
class DebugClassifier(ImageClassifier):
    def on_validation_epoch_end(self):
        # Print all metrics currently in the Trainer's logged_metrics dict
        print("Logged metrics:", self.trainer.logged_metrics)
        # Output: {'train/loss_step': tensor(1.23), 'train/loss_epoch': tensor(1.15),
        #          'val/loss': tensor(0.98), 'val/acc': tensor(0.62)}
        # Key 'val_loss' is NOT present — only 'val/loss' is logged
        # ReduceLROnPlateau cannot find 'val_loss' — it never fires

# Step 3: check current learning rate
optimizer = trainer.optimizers[0]
current_lr = optimizer.param_groups[0]['lr']
print(f"Current LR after epoch {trainer.current_epoch}: {current_lr}")
# Current LR after epoch 50: 0.1  ← unchanged from initial value
# Expected after patience=5 with flat val/loss: LR ~0.001 (two reductions)

The failure is invisible in TensorBoard because TensorBoard correctly shows the metric under the val/ group (as the developer intended), and the validation loss curve does show improvement. The developer has no reason to suspect the scheduler is broken — they see a reasonable training run with improving validation loss. The learning rate schedule not firing is the only signal, and it requires actively logging and monitoring the current LR value (which the developer had not configured) to notice.

The fix: use a shared MONITOR_KEY constant and add a LearningRateMonitor callback

Replace the hardcoded monitor='val_loss' string with a module-level constant and use the same constant in validation_step(). Add a LearningRateMonitor callback to log the current learning rate to TensorBoard, and add an assertion in on_validation_epoch_end() that the monitor metric is present in logged metrics after every validation epoch.

# model.py — FIXED (Westfield Model Training Ltd.)
# pytorch-lightning 2.2.x, Python 3.11, PyTorch 2.3.x

import torch
import torch.nn as nn
import pytorch_lightning as pl
from torch.optim.lr_scheduler import ReduceLROnPlateau
from pytorch_lightning.callbacks import LearningRateMonitor, ModelCheckpoint

# Single source of truth for the monitored metric name
MONITOR_KEY = 'val_loss'   # used in both self.log() and configure_optimizers()

class ImageClassifier(pl.LightningModule):
    def __init__(self, num_classes: int, lr: float = 0.1):
        super().__init__()
        self.save_hyperparameters()
        self.model = resnet50(weights=None, num_classes=num_classes)
        self.criterion = nn.CrossEntropyLoss()

    def training_step(self, batch, batch_idx):
        x, y = batch
        logits = self.model(x)
        loss = self.criterion(logits, y)
        self.log('train_loss', loss, on_step=True, on_epoch=True, prog_bar=True)
        return loss

    def validation_step(self, batch, batch_idx):
        x, y = batch
        logits = self.model(x)
        loss = self.criterion(logits, y)
        acc = (logits.argmax(dim=1) == y).float().mean()
        # Use the shared MONITOR_KEY constant — exact match for configure_optimizers()
        self.log(MONITOR_KEY, loss, on_epoch=True, sync_dist=True)
        self.log('val_acc',  acc,  on_epoch=True, sync_dist=True, prog_bar=True)

    def on_validation_epoch_end(self):
        # Assertion: verify monitor metric is present in logged metrics
        # Catches key mismatches at epoch 1 rather than after 50 epochs
        logged = self.trainer.logged_metrics
        assert MONITOR_KEY in logged, (
            f"Monitor metric '{MONITOR_KEY}' not found in logged metrics {list(logged.keys())}. "
            f"ReduceLROnPlateau and ModelCheckpoint will not work correctly. "
            f"Ensure self.log('{MONITOR_KEY}', ...) is called in validation_step()."
        )

    def configure_optimizers(self):
        optimizer = torch.optim.SGD(
            self.parameters(),
            lr=self.hparams.lr,
            momentum=0.9,
            weight_decay=1e-4,
        )
        scheduler = ReduceLROnPlateau(
            optimizer,
            mode='min',
            factor=0.1,
            patience=5,
            min_lr=1e-6,
        )
        return {
            'optimizer': optimizer,
            'lr_scheduler': {
                'scheduler': scheduler,
                'monitor': MONITOR_KEY,   # ← same constant as self.log() in validation_step()
                'interval': 'epoch',
                'frequency': 1,
            },
        }

# Trainer setup with LearningRateMonitor for visible LR tracking
trainer = pl.Trainer(
    max_epochs=50,
    callbacks=[
        LearningRateMonitor(logging_interval='epoch'),  # logs LR to TensorBoard
        ModelCheckpoint(
            monitor=MONITOR_KEY,   # same constant
            mode='min',
            save_top_k=3,
            filename='epoch{epoch:02d}-val_loss{val_loss:.4f}',
        ),
    ],
)

Validation

# Validation log — November 2026
# pytorch-lightning 2.2.x, PyTorch 2.3.x, SGD + ReduceLROnPlateau

# ---- Before fix ----
# configure_optimizers() monitor: 'val_loss'
# validation_step() self.log() key: 'val/loss'
# Lightning exact match: FAIL — 'val_loss' != 'val/loss'
# ReduceLROnPlateau: never stepped
# LR after epoch 50: 0.1 (unchanged)
# Validation loss after epoch 50: 0.31 (still decreasing slowly)
# Expected if LR had reduced: ~0.18 (faster convergence with LR schedule)

# ---- After fix: MONITOR_KEY = 'val_loss' used consistently ----
# configure_optimizers() monitor: 'val_loss'
# validation_step() self.log() key: 'val_loss'   ← consistent
# Lightning exact match: OK — 'val_loss' found in logged_metrics  ✓
# on_validation_epoch_end() assertion: PASSED at epoch 1            ✓
# LearningRateMonitor: logging LR to TensorBoard as 'lr-SGD'        ✓

# ---- Learning rate schedule after fix ----
# Epoch 1-5: LR = 0.1 (patience=5, val_loss improving)
# Epoch 6:   LR = 0.1 → 0.01 (val_loss plateaued at 0.42 for 5 epochs)  ✓
# Epoch 7-11: LR = 0.01 (improving again)
# Epoch 15:  LR = 0.01 → 0.001 (second plateau)                           ✓

# ---- Convergence comparison ----
# Without fix (flat LR=0.1, 50 epochs): val_loss = 0.31, val_acc = 0.887
# With fix (ReduceLROnPlateau active, 50 epochs): val_loss = 0.18, val_acc = 0.937 ✓

# ---- Hour breakdown ----
# Triage:     0.5h — LR flat at 0.1 for 50 epochs; val/loss improving;
#             confirm ReduceLROnPlateau not firing; enable DEBUG logging;
#             find 'Metric val_loss is not in logged metrics' message
# Root cause: 0.25h — compare monitor='val_loss' to self.log('val/loss');
#             confirm exact key mismatch; print logged_metrics dict
# Fix:        0.5h — add MONITOR_KEY constant; update self.log() key;
#             add LearningRateMonitor; add on_validation_epoch_end() assertion
# Validation: 0.25h — LR decreases at epoch 6; assertion passes; val_acc 0.887→0.937
# Total:      1.5h

# Retainer engagement cost: 1.5h (estimated $120–180 at mid-level rates)

Related retainer posts

• DeepSpeed developer on retainer — ZeRO Stage 2 custom checkpoint path without integer tag — optimizer state silently not loaded — training resumes from scratch — 3 weeks
• Hugging Face Accelerate developer on retainer — distributed BatchNorm running stats captured from rank 0 only — 1/4 of training batches — inference BN stats biased — 3 weeks
• XGBoost developer on retainer — unseen categorical value mapped to -1 at inference — all new-category records routed to same branch — false-positive rate 8%→31% — 3 weeks