Triton Inference Server developer on retainer: model config dims: [512] omits batch dimension — dynamic batching assembles 8–32 request groups but model processes each as shape [1, 512] — GPU utilization 15% under 64 concurrent clients — 3 weeks — Triton developer on monthly retainer

triton-inference-server 24.x • Python 3.11 • NVIDIA A100 • PyTorch TorchScript backend • BERT-base-uncased classification • config.pbtxt dims: [512] • dynamic batching: max_batch_size: 64, preferred_batch_size: [8, 16, 32] • each request processed as [1, 512] not [N, 512] • GPU utilization 15% • developer checks max_batch_size, preferred_batch_size, max_batch_delay_microseconds — 3 weeks • fix: stack inputs across requests in execute() • GPU utilization 15%→72% • 1.5h retainer engagement

Northfield Triton Serving Ltd. deployed a BERT-base-uncased text classification model on NVIDIA Triton Inference Server 24.x using a PyTorch TorchScript backend on an NVIDIA A100. They configured dynamic batching with max_batch_size: 64 and preferred_batch_size: [8, 16, 32] to improve throughput. During load testing with 64 concurrent clients sending individual classification requests, GPU utilization stayed at 15%. Single-request latency was acceptable, but throughput did not scale. The developer spent three weeks reviewing the dynamic batching configuration: adjusting max_batch_delay_microseconds, changing preferred_batch_size values, modifying the instance group count, and comparing GPU metrics before engaging a Triton retainer developer.

The model config and handler at the time of the engagement:

# config.pbtxt — Northfield Triton Serving Ltd.
# triton-inference-server 24.x, NVIDIA A100
# PyTorch TorchScript backend, BERT-base-uncased classification

name: "bert_classifier"
backend: "pytorch"
max_batch_size: 64

dynamic_batching {
  preferred_batch_size: [ 8, 16, 32 ]
  max_queue_delay_microseconds: 5000
}

input [
  {
    name: "input_ids"
    data_type: TYPE_INT64
    dims: [ 512 ]   # BUG: specifies 1D vector of 512 elements
                    # For PyTorch backend with max_batch_size > 0,
                    # this is correct — Triton prepends the batch dim
                    # BUT the Python model's execute() must stack
                    # per-request tensors manually.
  },
  {
    name: "attention_mask"
    data_type: TYPE_INT64
    dims: [ 512 ]
  },
  {
    name: "token_type_ids"
    data_type: TYPE_INT64
    dims: [ 512 ]
  }
]

output [
  {
    name: "logits"
    data_type: TYPE_FP32
    dims: [ 2 ]
  }
]
# model.py — PyTorch backend handler (broken)
# Northfield Triton Serving Ltd., triton 24.x

import numpy as np
import triton_python_backend_utils as pb_utils
import torch


class TritonPythonModel:

    def initialize(self, args):
        self.device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
        self.model = torch.jit.load("/models/bert_classifier/1/model.pt")
        self.model.to(self.device)
        self.model.eval()

    def execute(self, requests):
        responses = []
        for request in requests:
            # BUG: processes each request individually in a loop
            # Triton sends requests as a list — the developer iterates and
            # infers one sample at a time. Each GPU call processes shape [1, 512].
            # Dynamic batching assembled 16 requests into this execute() call,
            # but the loop sends 16 sequential single-sample forward passes.
            input_ids = pb_utils.get_input_tensor_by_name(request, "input_ids")
            attention_mask = pb_utils.get_input_tensor_by_name(request, "attention_mask")
            token_type_ids = pb_utils.get_input_tensor_by_name(request, "token_type_ids")

            input_ids_t = torch.tensor(input_ids.as_numpy(), device=self.device)
            attention_mask_t = torch.tensor(attention_mask.as_numpy(), device=self.device)
            token_type_ids_t = torch.tensor(token_type_ids.as_numpy(), device=self.device)

            # Shape: [1, 512] — single sample per GPU forward pass
            with torch.no_grad():
                logits = self.model(input_ids_t, attention_mask_t, token_type_ids_t)

            out_tensor = pb_utils.Tensor("logits", logits.cpu().numpy())
            responses.append(pb_utils.InferenceResponse(output_tensors=[out_tensor]))

        return responses

# GPU forward passes per execute() call with preferred_batch_size=[8,16,32]:
# Dynamic batching groups 16 requests → execute() called with len(requests)=16
# Loop: 16 sequential model() calls, each with shape [1, 512]
# GPU processes 16 single-sample matrices instead of one [16, 512] matrix
# GPU utilization: 15% (GPU kernel launch overhead dominates; CUDA cores idle)

Why dynamic batching metrics look correct while GPU utilization stays at 15%

Triton’s dynamic batching assembles requests into groups based on preferred_batch_size and max_queue_delay_microseconds. The Triton Prometheus metrics show nv_inference_request_success increasing and batch size histograms reporting values of 8–32 as configured. From Triton’s perspective, batching is working: it is grouping requests and dispatching each group to the model backend as a single execute() call with len(requests) == preferred_batch_size.

The issue is inside execute(). The Python backend handler receives a list of N request objects—one per request in the assembled batch. Each request object contains the tensor data for one sample. If the handler iterates over the request list and runs model(input_ids_t, attention_mask_t, token_type_ids_t) once per loop iteration, each GPU call processes a single sample of shape [1, 512]. Triton’s batching has no effect on the GPU computation: the assembling is done, but the handler re-serializes the batch back into sequential single-sample forward passes.

The GPU kernel launch overhead for a [1, 512] BERT-base forward pass is nearly identical to a [32, 512] forward pass (both require the same number of CUDA kernel launches), but the compute work for the [32, 512] case is 32 times larger. Running 32 sequential [1, 512] forward passes instead of one [32, 512] forward pass introduces 32x the kernel launch overhead while performing the same total compute—but not in parallel. GPU utilization, measured as the fraction of time the CUDA cores are executing tensor operations, stays low because the GPU spends most of its time in kernel launch overhead and memory staging between sequential calls.

# Diagnosing the batch dimension issue
# Add shape logging to execute() to inspect what the GPU is receiving

import numpy as np
import triton_python_backend_utils as pb_utils
import torch


class TritonPythonModel:

    def initialize(self, args):
        self.device = torch.device("cuda")
        self.model = torch.jit.load("/models/bert_classifier/1/model.pt")
        self.model.to(self.device)
        self.model.eval()
        self._call_count = 0

    def execute(self, requests):
        # DIAGNOSTIC: log how many requests arrived and what shape each tensor is
        self._call_count += 1
        print(f"execute() call #{self._call_count}: len(requests) = {len(requests)}")

        if len(requests) > 0:
            sample_input = pb_utils.get_input_tensor_by_name(requests[0], "input_ids")
            sample_array = sample_input.as_numpy()
            print(f"  requests[0] input_ids numpy shape: {sample_array.shape}")
            # Output: requests[0] input_ids numpy shape: (1, 512)
            # This confirms each request holds one sample of shape [1, 512]
            # even when 16 requests were assembled by dynamic batching.

        # The fix: stack all request tensors into a single batch tensor
        # before the GPU forward pass.
        all_input_ids = np.vstack([
            pb_utils.get_input_tensor_by_name(r, "input_ids").as_numpy()
            for r in requests
        ])
        print(f"  stacked input_ids shape: {all_input_ids.shape}")
        # Output: stacked input_ids shape: (16, 512)
        # Shape is now [batch_size, seq_len] — one batched GPU call

        return []  # diagnostic only, no real responses here

The fix: stack all request tensors before the GPU forward pass

The fix replaces the per-request loop with a single batched GPU forward pass. All input tensors from the assembled requests are stacked into a [N, 512] batch before calling model(). The GPU processes all N samples in one parallel forward pass. The logits tensor is then split back into per-request response objects.

# model.py — FIXED (Northfield Triton Serving Ltd.)
# triton-inference-server 24.x, Python 3.11, NVIDIA A100

import numpy as np
import triton_python_backend_utils as pb_utils
import torch


class TritonPythonModel:

    def initialize(self, args):
        self.device = torch.device("cuda")
        self.model = torch.jit.load("/models/bert_classifier/1/model.pt")
        self.model.to(self.device)
        self.model.eval()

    def execute(self, requests):
        # FIXED: stack all request tensors into a single batched tensor
        # Dynamic batching assembled len(requests) samples — process them together
        batch_size = len(requests)

        all_input_ids = np.vstack([
            pb_utils.get_input_tensor_by_name(r, "input_ids").as_numpy()
            for r in requests
        ])  # shape: [batch_size, 512]
        all_attention_mask = np.vstack([
            pb_utils.get_input_tensor_by_name(r, "attention_mask").as_numpy()
            for r in requests
        ])  # shape: [batch_size, 512]
        all_token_type_ids = np.vstack([
            pb_utils.get_input_tensor_by_name(r, "token_type_ids").as_numpy()
            for r in requests
        ])  # shape: [batch_size, 512]

        input_ids_t = torch.tensor(all_input_ids, dtype=torch.long, device=self.device)
        attention_mask_t = torch.tensor(all_attention_mask, dtype=torch.long, device=self.device)
        token_type_ids_t = torch.tensor(all_token_type_ids, dtype=torch.long, device=self.device)

        # Single batched GPU forward pass: shape [batch_size, 512]
        # GPU processes all samples in parallel — kernel launched once for N samples
        with torch.no_grad():
            logits = self.model(input_ids_t, attention_mask_t, token_type_ids_t)
        # logits shape: [batch_size, 2]

        # Assertion: verify expected output batch dimension
        assert logits.shape[0] == batch_size, (
            f"Expected logits.shape[0] == {batch_size}, got {logits.shape[0]}"
        )

        # Split back into per-request responses
        logits_np = logits.cpu().numpy()  # [batch_size, 2]
        responses = []
        for i in range(batch_size):
            out_tensor = pb_utils.Tensor("logits", logits_np[i:i+1])  # [1, 2]
            responses.append(pb_utils.InferenceResponse(output_tensors=[out_tensor]))

        return responses

# After fix with preferred_batch_size=[8,16,32] and 64 concurrent clients:
# Each execute() call processes 8-32 samples in one GPU forward pass
# GPU utilization: 15% → 72%
# Throughput (requests/sec): 48 → 312 (6.5x improvement)
# p99 latency: 140ms → 18ms

Validation

# Validation log — November 2026
# triton-inference-server 24.x, BERT-base-uncased, NVIDIA A100
# PyTorch TorchScript backend, max_batch_size=64, preferred_batch_size=[8,16,32]

# ---- Before fix (per-request loop in execute(), sequential GPU forward passes) ----
# 1 concurrent client:   GPU utilization  7%  — 1 sample per GPU call as expected
# 8 concurrent clients:  GPU utilization 12%  — 8 sequential [1,512] calls per execute()
# 32 concurrent clients: GPU utilization 14%  — 32 sequential calls per execute()
# 64 concurrent clients: GPU utilization 15%  — 32-64 sequential calls per execute()
# Throughput @ 64 clients: 48 requests/sec
# p99 latency @ 64 clients: 140ms

# ---- After fix (stacked batch tensor, single GPU forward pass per execute()) ----
# 1 concurrent client:   GPU utilization  8%  — same (batch_size=1 still a single call)
# 8 concurrent clients:  GPU utilization 41%  — one [8,512] GPU forward pass
# 32 concurrent clients: GPU utilization 68%  — one [32,512] GPU forward pass
# 64 concurrent clients: GPU utilization 72%  — one [64,512] GPU forward pass
# Throughput @ 64 clients: 312 requests/sec  (+550%)
# p99 latency @ 64 clients: 18ms  (-87%)

# Test set: 64 concurrent clients, 1000 requests each, BERT classification
# GPU utilization under load: 72% (was 15%)                          ✓
# Throughput: 312 req/sec (was 48)                                   ✓
# Classification accuracy unchanged (same model weights)             ✓
# Per-request output shape matches pre-fix: [1, 2] logits per sample ✓

# ---- Hour breakdown ----
# Triage:     0.5h — GPU at 15% under 64 concurrent clients;
#             add shape logging to execute(); confirm [1,512] per call;
#             discover per-request loop in execute(); root cause identified
# Root cause: 0.25h — sequential single-sample forward passes confirmed;
#             each execute() call does N sequential [1,512] GPU calls not 1 [N,512]
# Fix:        0.5h — stack request tensors before model(); single batched call;
#             split logits back to per-request responses; add batch assertion
# Validation: 0.25h — 64-client load test; GPU 15%→72%; throughput 48→312 req/s
# Total:      1.5h

# Retainer engagement cost: 1.5h (estimated $165–255 at mid-level rates)

Related retainer posts

Track retainer hours without a client portal

HourTab turns your time-tracker CSV into a public URL your client bookmarks. No login. No portal. CSV in, URL out.

See pricing →