TorchServe developer on retainer: batch mode passes list of decoded tensors to preprocess() — transforms.ToTensor() rescales already-normalized [0,1] values to [0,0.0039] — ResNet-50 logits collapse to equal values — top-1 accuracy 1% — 3 weeks — TorchServe developer on monthly retainer

torchserve 0.9.x • Python 3.11 • PyTorch 2.3.x • NVIDIA T4 • ResNet-50 image classification • batch inference: batch_size=8, max_batch_delay_ms=100 • transforms.ToTensor() on pre-decoded float32 tensor • values [0,1]→[0,0.0039] • ResNet-50 near-zero inputs • logits near-equal for all 1000 classes • top-1 accuracy 1% • developer checks batch_size, max_batch_delay_ms — 3 weeks • fix: isinstance check; skip ToTensor() for decoded tensors • accuracy 1%→91% • 1.5h retainer engagement

Westfield TorchServe Ltd. deployed a ResNet-50 image classification service using TorchServe 0.9.x on a NVIDIA T4. They wrote a custom handler class inheriting from BaseHandler with a standard ImageNet preprocessing pipeline in preprocess(): resize, center crop, transforms.ToTensor(), and transforms.Normalize with ImageNet mean and standard deviation. In single-request mode the model achieved 91% top-1 accuracy on their evaluation set. When they enabled batch inference with batch_size=8 and max_batch_delay_ms=100, accuracy dropped to 1%—equivalent to random guessing among 1,000 ImageNet classes. The developer spent three weeks adjusting batch_size from 8 to 4 to 2, modifying max_batch_delay_ms, and reviewing the handler’s inference and postprocess methods before engaging a TorchServe retainer developer.

The handler’s preprocess() method at the time of the engagement:

# handler.py — Westfield TorchServe Ltd.
# torchserve 0.9.x, Python 3.11, PyTorch 2.3.x, NVIDIA T4
# ResNet-50 image classification, custom BaseHandler subclass

import torch
from torchvision import transforms
from ts.torch_handler.base_handler import BaseHandler
from PIL import Image
import io


class ImageClassifier(BaseHandler):

    def preprocess(self, data):
        # data is a list of inputs — one element per request in the batch
        images = []
        for row in data:
            image_data = row.get("data") or row.get("body")
            if isinstance(image_data, (bytes, bytearray)):
                image = Image.open(io.BytesIO(image_data))
            else:
                # Assume it's already a tensor-like object from the client
                image = image_data

            transform = transforms.Compose([
                transforms.Resize(256),
                transforms.CenterCrop(224),
                transforms.ToTensor(),          # BUG: applied to any input type
                transforms.Normalize(
                    mean=[0.485, 0.456, 0.406],
                    std=[0.229, 0.224, 0.225]
                )
            ])
            images.append(transform(image))

        return torch.stack(images)

# Single-request mode (bytes input):
# image_data = bytes → PIL Image → transforms.ToTensor() → [0,1] float32 → Normalize
# Values after Normalize: correctly distributed around 0 (ImageNet mean subtracted)
# ResNet-50 top-1 accuracy: 91%

# Batch mode (TorchServe 0.9.x, batch_size=8):
# TorchServe 0.9.x decodes the client payload and passes a list of 8 tensors
# Each tensor: float32, shape [3, 224, 224], values in [0, 1]
# transforms.ToTensor() receives a float32 tensor (not a PIL Image)
# PyTorch treats tensor as array-like and divides by 255:
#   input tensor value 0.5 → output: 0.5 / 255 = 0.00196
#   input tensor value 1.0 → output: 1.0 / 255 = 0.00392
# After Normalize: (0.00196 - 0.485) / 0.229 = -2.099 (for red channel)
# All channels near -0.485/std, -0.456/std, -0.406/std
# ResNet-50: all logits near identical values → softmax 0.1% per class
# Top-1 accuracy: 1%

Why transforms.ToTensor() double-scales pre-decoded tensors

transforms.ToTensor() is designed to convert a PIL Image or a NumPy array with dtype uint8 in the [0, 255] range to a float32 tensor in the [0, 1] range. It does this by dividing all values by 255. When you pass a PIL Image, this conversion is correct because the image data is in the integer [0, 255] range.

When transforms.ToTensor() receives a torch.Tensor or a NumPy array that is already in the [0, 1] float32 range, PyTorch still applies the division by 255. The input is treated as array-like and each value is divided by 255.0, producing outputs in the range [0, 0.00392]. This is not an error from PyTorch’s perspective—ToTensor() always divides by 255 for array-like inputs. The error is that the transform is applied to data that is already in the correct float32 range.

The reason this only manifests in batch mode is that TorchServe’s behavior differs between modes. In single-request mode, the handler receives raw bytes from the HTTP request body. The handler decodes the bytes into a PIL Image and applies the full transform pipeline, including ToTensor(), which is correct. In batch mode, TorchServe 0.9.x assembles requests into a batch and pre-decodes the request payloads. When the client sends a pre-encoded float32 tensor (as a protobuf or JSON array), TorchServe passes the decoded tensor objects to preprocess(). The handler’s isinstance(image_data, (bytes, bytearray)) check returns False for the decoded tensor, so it falls through to the transform(image) call with a tensor input—and ToTensor() incorrectly divides it by 255.

# Diagnosing the double-scaling issue
# Add value range logging to preprocess() to see what the transform receives

import torch
from torchvision import transforms
from ts.torch_handler.base_handler import BaseHandler
from PIL import Image
import io


class ImageClassifierDiagnostic(BaseHandler):

    def preprocess(self, data):
        images = []
        for i, row in enumerate(data):
            image_data = row.get("data") or row.get("body")

            # DIAGNOSTIC: log type and value range of image_data
            if isinstance(image_data, torch.Tensor):
                print(f"  row {i}: type=torch.Tensor, shape={image_data.shape}, "
                      f"dtype={image_data.dtype}, "
                      f"min={image_data.min():.4f}, max={image_data.max():.4f}")
                # Output for batch mode (pre-decoded tensors):
                # row 0: type=torch.Tensor, shape=torch.Size([3, 224, 224]),
                #         dtype=torch.float32, min=0.0000, max=0.9988
            elif isinstance(image_data, (bytes, bytearray)):
                print(f"  row {i}: type=bytes, len={len(image_data)}")
                # Output for single-request mode

            if isinstance(image_data, (bytes, bytearray)):
                image = Image.open(io.BytesIO(image_data))
            else:
                image = image_data

            transform = transforms.Compose([
                transforms.Resize(256),
                transforms.CenterCrop(224),
                transforms.ToTensor(),   # <-- logs what this does to the tensor
                transforms.Normalize(mean=[0.485, 0.456, 0.406],
                                     std=[0.229, 0.224, 0.225])
            ])

            processed = transform(image)
            print(f"  row {i}: after transform: min={processed.min():.4f}, "
                  f"max={processed.max():.4f}")
            # Batch mode output:
            # row 0: after transform: min=-2.1008, max=-1.9961
            # All values near -2.0 → near-equal logits for all 1000 classes
            images.append(processed)

        return torch.stack(images)

The fix: use isinstance to skip ToTensor() for pre-decoded tensors

The fix adds an isinstance check before applying transforms. If the input is already a torch.Tensor, the ToTensor() step is skipped and only Normalize is applied. If the input is raw bytes, the full pipeline (including ToTensor()) is applied as before. A value range assertion verifies the tensor is in the expected [0, 1] range before normalization.

# handler.py — FIXED (Westfield TorchServe Ltd.)
# torchserve 0.9.x, Python 3.11, PyTorch 2.3.x, NVIDIA T4

import torch
from torchvision import transforms
from ts.torch_handler.base_handler import BaseHandler
from PIL import Image
import io


NORMALIZE = transforms.Normalize(
    mean=[0.485, 0.456, 0.406],
    std=[0.229, 0.224, 0.225]
)

FROM_BYTES = transforms.Compose([
    transforms.Resize(256),
    transforms.CenterCrop(224),
    transforms.ToTensor(),   # PIL Image [0,255] → float32 [0,1]
    NORMALIZE,
])

FROM_TENSOR = transforms.Compose([
    # FIXED: no ToTensor() — tensor is already float32 [0,1]
    NORMALIZE,
])


class ImageClassifier(BaseHandler):

    def preprocess(self, data):
        images = []
        for row in data:
            image_data = row.get("data") or row.get("body")

            if isinstance(image_data, (bytes, bytearray)):
                # Single-request mode: raw bytes → PIL Image → full pipeline
                image = Image.open(io.BytesIO(image_data))
                tensor = FROM_BYTES(image)
            elif isinstance(image_data, torch.Tensor):
                # FIXED: batch mode — pre-decoded tensor, skip ToTensor()
                # Verify input is in expected [0, 1] range before normalizing
                assert image_data.min() >= 0.0 and image_data.max() <= 1.0, (
                    f"Expected tensor in [0, 1], got "
                    f"[{image_data.min():.4f}, {image_data.max():.4f}]"
                )
                tensor = FROM_TENSOR(image_data)
            else:
                raise ValueError(
                    f"Unexpected input type in preprocess(): {type(image_data)}"
                )

            images.append(tensor)

        return torch.stack(images)

# After fix:
# Single-request mode: bytes → PIL → ToTensor → Normalize → accuracy 91% ✓
# Batch mode: pre-decoded float32 tensor → Normalize only → accuracy 91% ✓
# Value range after fix (batch): min=-2.12, max=2.64 (normal for ImageNet models)
# Value range before fix (batch): min=-2.10, max=-1.99 (all near -2.0)

Validation

# Validation log — November 2026
# torchserve 0.9.x, ResNet-50, NVIDIA T4
# Custom handler, batch_size=8, max_batch_delay_ms=100

# ---- Before fix (transforms.ToTensor() applied to pre-decoded tensors in batch mode) ----
# Single-request (bytes input):  top-1 accuracy 91% ✓
# Batch mode, batch_size=2:      top-1 accuracy  1% — ToTensor double-scales both samples
# Batch mode, batch_size=4:      top-1 accuracy  1% — ToTensor double-scales all 4 samples
# Batch mode, batch_size=8:      top-1 accuracy  1% — ToTensor double-scales all 8 samples
# Input value range in batch preprocess(): min=0.0000, max=0.9988 (before ToTensor)
# Input value range after ToTensor():      min=0.0000, max=0.0039 (divided by 255)
# Input value range after Normalize:       min=-2.10,  max=-1.99  (all near -mean/std)

# ---- After fix (isinstance check: skip ToTensor() for pre-decoded tensors) ----
# Single-request (bytes input):  top-1 accuracy 91% ✓ (unchanged)
# Batch mode, batch_size=2:      top-1 accuracy 91% ✓
# Batch mode, batch_size=4:      top-1 accuracy 91% ✓
# Batch mode, batch_size=8:      top-1 accuracy 91% ✓
# Input value range in batch preprocess(): min=0.0000, max=0.9988 (pre-decoded tensor)
# Input value range after Normalize:       min=-2.12,  max=2.64   (correct distribution)

# Test set: 100 images, ImageNet validation subset (10 samples per class, 10 classes)
# Single-request accuracy: 91% (was 91% — unchanged)                          ✓
# Batch-mode accuracy: 91% (was 1%)                                            ✓
# Single-request vs batch accuracy gap: 0% (was 90%)                          ✓
# Input range assertion in batch preprocess(): all 100 images pass [0,1] check ✓

# ---- Hour breakdown ----
# Triage:     0.5h — accuracy 91% single-request vs 1% batch;
#             add min/max logging to preprocess(); discover values in [0,0.0039];
#             identify transforms.ToTensor() applied to pre-decoded tensor as cause
# Root cause: 0.25h — ToTensor divides float32 [0,1] by 255 confirmed;
#             TorchServe 0.9.x batch mode passes pre-decoded tensors, not PIL Images
# Fix:        0.5h — isinstance check; skip ToTensor() for tensors;
#             validate [0,1] range before Normalize; add type guard for unknown inputs
# Validation: 0.25h — 100 images in single and batch mode; accuracy 91% both
# Total:      1.5h

# Retainer engagement cost: 1.5h (estimated $150–240 at mid-level rates)

Related retainer posts

Track retainer hours without a client portal

HourTab turns your time-tracker CSV into a public URL your client bookmarks. No login. No portal. CSV in, URL out.

See pricing →