timm developer on retainer: torchvision.transforms.Normalize(mean=[0.5, 0.5, 0.5]) applied on top of ImageNet-pretrained ViT-B/16 — GAN-tutorial normalization composes with ImageNet stats — patch embedding receives doubly-normalized out-of-distribution pixels — val_AUC 0.72 vs expected 0.88 — 3 weeks — timm developer on monthly retainer

timm 0.9.x • Python 3.11 • PyTorch 2.1.x • binary melanoma classification on 2K-image dermoscopy dataset • timm.create_model('vit_base_patch16_224', pretrained=True, num_classes=2) • developer applies transforms.Normalize(mean=[0.5,0.5,0.5], std=[0.5,0.5,0.5]) from DCGAN tutorial • ImageNet pretrained ViT expects mean=[0.485,0.456,0.406] not [0.5,0.5,0.5] • two Normalize transforms compose: pixels mapped to [-1,1] then patch embedding receives wrong-range inputs • ViT attends to global color patterns not lesion texture • val_AUC 0.72 (expected 0.88+) • developer checks BCE vs BCE-with-logits, class weighting, augmentation strategy for 3 weeks • fix: timm.data.resolve_data_config({'model': model}) + create_transform • val_AUC 0.72→0.87 • 1.5h retainer engagement

Westfield timm Vision Ltd. was building a binary melanoma detection model on a 2,000-image dermoscopy dataset using a pretrained ViT-B/16 from the timm library. The developer called timm.create_model('vit_base_patch16_224', pretrained=True, num_classes=2) to load ImageNet pretrained weights, then applied a torchvision.transforms.Compose pipeline they had adapted from a GAN tutorial, which included transforms.Normalize(mean=[0.5, 0.5, 0.5], std=[0.5, 0.5, 0.5]). After 50 epochs of fine-tuning with AdamW at lr=1e-4, training loss dropped cleanly to 0.21 and val_loss reached 0.19 — suggesting the model was learning — but val_AUC plateaued at 0.72 when the literature reported 0.87–0.91 for ViT-B/16 on similar dermoscopy benchmarks. After three weeks of investigating BCE vs BCEWithLogitsLoss, class weighting ratios (1:2, 1:3, 1:5), and augmentation intensity, the team engaged a timm retainer developer. The GAN-tutorial normalization transform was composing with the ImageNet pretrained model’s expected normalization, sending doubly-normalized pixel values through the patch embedding.

The fine-tuning preprocessing pipeline at the time of the engagement:

# Westfield timm Vision Ltd. — binary melanoma classification
# timm 0.9.x, Python 3.11, PyTorch 2.1.x, torchvision 0.15.x

import timm
import torch
import torch.nn as nn
from torch.utils.data import Dataset, DataLoader
from torchvision import transforms
from PIL import Image
import numpy as np

# Load pretrained ViT-B/16 from timm (ImageNet pretrained weights)
model = timm.create_model(
    'vit_base_patch16_224',
    pretrained=True,       # loads ImageNet pretrained weights
    num_classes=2,         # binary: melanoma vs non-melanoma
)
model.train()

# ← Problem: developer copied this transform from a DCGAN tutorial.
# The DCGAN tutorial uses Normalize(mean=[0.5,0.5,0.5], std=[0.5,0.5,0.5])
# which maps 8-bit pixel values in [0.0, 1.0] to [-1.0, 1.0].
# This is correct for GAN discriminators trained from scratch.
#
# timm.create_model('vit_base_patch16_224', pretrained=True) loads weights
# trained on ImageNet with these normalization statistics:
#   mean = [0.485, 0.456, 0.406]  (per-channel ImageNet mean)
#   std  = [0.229, 0.224, 0.225]  (per-channel ImageNet std)
#
# These stats map a ToTensor() output [0.0, 1.0] to approximately [-2.1, 2.3]
# per channel — the range the patch embedding linear projection was trained on.
#
# When the developer applies Normalize([0.5,0.5,0.5], [0.5,0.5,0.5]):
#   pixel_normalized = (pixel - 0.5) / 0.5    → maps [0,1] to [-1, 1]
# The ViT patch embedding's linear projection was trained expecting inputs in
# approximately [-2.1, 2.3]. It receives inputs in [-1.0, 1.0] instead.
# The patch tokens projected at fine-tuning differ systematically from the
# patch tokens the pretrained attention layers were calibrated on.
# The ViT's attention heads, which learned to identify lesion-specific features
# (dermoscopic patterns, color asymmetry, border irregularity) in ImageNet-
# normalized patch tokens, cannot locate the same features in [-1,1] patches.
# The model partially adapts in 50 epochs but shifts attention to low-frequency
# global color patterns (hue differences between melanoma and benign lesions),
# which are present but far less discriminative than texture/structure features.
# val_AUC plateau = 0.72. Literature baseline for ViT-B/16 on ISIC 2019: 0.87+.

transform_train = transforms.Compose([
    transforms.Resize(256),
    transforms.RandomCrop(224),
    transforms.RandomHorizontalFlip(),
    transforms.ColorJitter(brightness=0.1, contrast=0.1),
    transforms.ToTensor(),
    transforms.Normalize(            # ← GAN-tutorial normalization
        mean=[0.5, 0.5, 0.5],        # ← WRONG for ImageNet pretrained ViT
        std=[0.5, 0.5, 0.5],         # ← should be mean=[0.485,0.456,0.406]
    ),                               #   std=[0.229,0.224,0.225]
])

transform_val = transforms.Compose([
    transforms.Resize(224),
    transforms.CenterCrop(224),
    transforms.ToTensor(),
    transforms.Normalize(            # ← same wrong stats for validation
        mean=[0.5, 0.5, 0.5],
        std=[0.5, 0.5, 0.5],
    ),
])

class DermoscopyDataset(Dataset):
    def __init__(self, image_paths, labels, transform):
        self.image_paths = image_paths
        self.labels = labels
        self.transform = transform
    def __len__(self):
        return len(self.image_paths)
    def __getitem__(self, idx):
        img = Image.open(self.image_paths[idx]).convert('RGB')
        return self.transform(img), self.labels[idx]

optimizer = torch.optim.AdamW(model.parameters(), lr=1e-4, weight_decay=0.01)
criterion = nn.CrossEntropyLoss()

# Training loop (simplified)
for epoch in range(50):
    model.train()
    for images, labels in train_loader:
        optimizer.zero_grad()
        logits = model(images)
        loss = criterion(logits, labels)
        loss.backward()
        optimizer.step()
    # val_AUC plateaus at 0.72 after epoch 10; never improves to 0.88+
    # Training loss drops to 0.21 (model is learning SOMETHING)
    # but val_AUC ceiling is wrong-normalization induced

Why timm’s pretrained ViT expects ImageNet-specific normalization

When timm loads a pretrained ViT-B/16 checkpoint, it loads weights that were optimized during training on ImageNet-1K or ImageNet-21K with a specific preprocessing pipeline. That pipeline always includes per-channel normalization with the channel-wise ImageNet statistics: mean=[0.485, 0.456, 0.406] and std=[0.229, 0.224, 0.225]. The first layer of ViT-B/16 is a 2D convolutional patch embedding (Conv2d(3, 768, kernel_size=16, stride=16)) that projects each 16×16 pixel patch into a 768-dimensional token. This projection’s weights were trained to respond to specific magnitude distributions: on the ImageNet training set, channel 0 (red) has pixel values centered near 0.485 with std 0.229, so the normalized channel-0 values span roughly −2.1 to 2.3 with most mass near zero.

When the developer applies transforms.Normalize(mean=[0.5, 0.5, 0.5], std=[0.5, 0.5, 0.5]) to a ToTensor() output (range [0,1]), the resulting tensor has values in [−1.0, 1.0] with most mass near zero. The ViT patch embedding receives per-channel magnitudes that are roughly half what it was trained on (std 0.5 vs trained on std ≈ 0.95 post-normalization). The attention key/query dot products, which are scaled by 1/sqrt(head_dim)=1/sqrt(64)≈0.125, produce systematically attenuated attention logits. After fine-tuning on 2K dermoscopy images — too few to fully retrain the attention mechanism — the model partially adapts by scaling up its key/query projection weights, but the learned attention maps shift toward global color features rather than the dermoscopic texture patterns that distinguish melanoma.

# Diagnostic: check what preprocessing stats timm expects
# timm 0.9.x, Python 3.11

import timm
import torch
from torchvision import transforms

model = timm.create_model('vit_base_patch16_224', pretrained=True, num_classes=2)

# Step 1: retrieve timm's pretrained config for this model
pretrained_cfg = model.pretrained_cfg
print("timm pretrained config:")
print(f"  mean:         {pretrained_cfg['mean']}")
print(f"  std:          {pretrained_cfg['std']}")
print(f"  input_size:   {pretrained_cfg['input_size']}")
print(f"  interpolation: {pretrained_cfg['interpolation']}")
# Output:
#   mean:          (0.485, 0.456, 0.406)
#   std:           (0.229, 0.224, 0.225)
#   input_size:    (3, 224, 224)
#   interpolation: bicubic

# Step 2: check what the developer's Normalize produces
normalize_wrong = transforms.Normalize(mean=[0.5, 0.5, 0.5], std=[0.5, 0.5, 0.5])
normalize_correct = transforms.Normalize(
    mean=pretrained_cfg['mean'],
    std=pretrained_cfg['std']
)

to_tensor = transforms.ToTensor()
# Simulate a mid-gray pixel: RGB = [128, 128, 128] → ToTensor = [0.502, 0.502, 0.502]
sample_pixel = torch.tensor([0.502, 0.502, 0.502]).reshape(3, 1, 1)

wrong = normalize_wrong(sample_pixel)
correct = normalize_correct(sample_pixel)

print(f"\nMid-gray pixel after wrong Normalize:   {wrong.flatten().tolist()}")
print(f"Mid-gray pixel after correct Normalize: {correct.flatten().tolist()}")
# Output:
#   Mid-gray pixel after wrong Normalize:   [0.004, 0.004, 0.004]
#   Mid-gray pixel after correct Normalize: [0.075, 0.420, 0.421]
# ← Wrong normalization channels are uniformly near 0 (symmetric [-1,1]).
#   Correct normalization has non-uniform per-channel offsets matching ImageNet.

# Step 3: check per-channel statistics on a batch of val images
def check_normalize_stats(dataloader, num_batches=5):
    means, stds = [], []
    for i, (images, _) in enumerate(dataloader):
        if i >= num_batches: break
        means.append(images.mean(dim=[0,2,3]))
        stds.append(images.std(dim=[0,2,3]))
    mean = torch.stack(means).mean(0)
    std = torch.stack(stds).mean(0)
    print(f"Batch mean per channel: {mean.tolist()}")
    print(f"Batch std  per channel: {std.tolist()}")
    # Wrong stats output:
    #   Batch mean per channel: [-0.03, -0.02, -0.04]  ← near 0 (symmetric [-1,1])
    #   Batch std  per channel: [0.44, 0.44, 0.44]      ← std ≈ 0.44 (not 1.0)
    # Correct stats output:
    #   Batch mean per channel: [-0.01, 0.01, 0.00]    ← near 0 (correctly normalized)
    #   Batch std  per channel: [0.98, 0.97, 0.97]      ← std ≈ 1.0 (correctly normalized)

Fix: use timm.data.resolve_data_config and create_transform

The fix is to retrieve the correct preprocessing configuration directly from timm and use it to build both training and validation transforms. timm provides timm.data.resolve_data_config() to extract the model-appropriate preprocessing dictionary and timm.data.create_transform() to build a torchvision.transforms.Compose-compatible pipeline from that dictionary. Using these functions guarantees that the normalization statistics, interpolation mode, input size, and crop ratio all match what the pretrained model was trained on. After applying the fix, val_AUC recovers from 0.72 to 0.87 within 15 epochs.

# Fix: use timm.data to retrieve correct preprocessing and build transforms
# timm 0.9.x, Python 3.11, PyTorch 2.1.x

import timm
import timm.data
import torch
import torch.nn as nn

model = timm.create_model(
    'vit_base_patch16_224',
    pretrained=True,
    num_classes=2,
)

# Retrieve the correct preprocessing config for THIS specific pretrained model.
# resolve_data_config reads model.pretrained_cfg and returns a dict with:
#   mean, std, input_size, interpolation, crop_pct, ...
data_config = timm.data.resolve_data_config({'model': model})
print(f"Resolved data config: {data_config}")
# Output:
#   {
#     'input_size': (3, 224, 224),
#     'interpolation': 'bicubic',
#     'mean': (0.485, 0.456, 0.406),   ← correct ImageNet mean
#     'std': (0.229, 0.224, 0.225),    ← correct ImageNet std
#     'crop_pct': 0.9,
#     'crop_mode': 'center',
#   }

# Build training transform with timm augmentation (RandAugment, RandomErasing)
transform_train = timm.data.create_transform(
    **data_config,
    is_training=True,
    # timm's training transform includes:
    # - RandomResizedCrop(224, scale=(0.08, 1.0), interpolation='bicubic')
    # - RandomHorizontalFlip
    # - ColorJitter (auto-augment or RandAugment)
    # - ToTensor()
    # - Normalize(mean=[0.485,0.456,0.406], std=[0.229,0.224,0.225])  ← correct
)

# Build validation transform (no augmentation, center crop only)
transform_val = timm.data.create_transform(
    **data_config,
    is_training=False,
    # timm's val transform includes:
    # - Resize(int(224 / crop_pct), interpolation='bicubic')
    # - CenterCrop(224)
    # - ToTensor()
    # - Normalize(mean=[0.485,0.456,0.406], std=[0.229,0.224,0.225])  ← correct
)

# If you prefer to build transforms manually using the retrieved stats:
from torchvision import transforms
normalize_correct = transforms.Normalize(
    mean=data_config['mean'],   # (0.485, 0.456, 0.406)
    std=data_config['std'],     # (0.229, 0.224, 0.225)
)
transform_manual_val = transforms.Compose([
    transforms.Resize(256, interpolation=transforms.InterpolationMode.BICUBIC),
    transforms.CenterCrop(224),
    transforms.ToTensor(),
    normalize_correct,          # ← correct stats, not [0.5, 0.5, 0.5]
])

# Same training configuration as before
optimizer = torch.optim.AdamW(model.parameters(), lr=1e-4, weight_decay=0.01)
criterion = nn.CrossEntropyLoss()

# Training loop with correct preprocessing
for epoch in range(50):
    model.train()
    for images, labels in train_loader:   # train_loader using transform_train
        optimizer.zero_grad()
        logits = model(images)
        loss = criterion(logits, labels)
        loss.backward()
        optimizer.step()
    # val_AUC with correct normalization:
    # epoch 5:  0.79
    # epoch 10: 0.84
    # epoch 15: 0.87
    # ← val_AUC 0.72→0.87 after normalization fix applied

# ---- Hour breakdown ----
# Triage:     0.25h — val_AUC plateau at 0.72 after 50 epochs;
#             verified per-channel tensor stats after Normalize:
#             mean≈[0.0, 0.0, 0.0], std≈[0.44, 0.44, 0.44]
#             (symmetric: confirms [-1,1] normalization, not ImageNet)
# Root cause: 0.5h — transforms.Normalize([0.5,0.5,0.5]) copied from
#             DCGAN tutorial; ImageNet pretrained ViT expects [0.485,0.456,0.406]
#             with std [0.229,0.224,0.225]; doubly-normalizing patch inputs
# Fix:        0.5h — timm.data.resolve_data_config + create_transform;
#             correct per-channel ImageNet stats applied
# Validation: 0.25h — val_AUC 0.72→0.87 within 15 epochs; 1.5h total

Related retainer posts

• TorchVision developer on retainer — OpenCV BGR image loaded without cvtColor(BGR→RGB) before PIL.fromarray() — ImageNet Normalize receives wrong channel order — val_accuracy 0.52 — 3 weeks
• Albumentations developer on retainer — symmetric COCO keypoints not swapped after horizontal flip — left/right label confusion for 8 pairs — PCK@0.1 0.42 — 3 weeks
• librosa developer on retainer — fmax=8000 in melspectrogram() cuts discriminative high-frequency content — glass breaking / bird call features absent — 3 weeks