BentoML developer on retainer: LightGBM runner initialized inside @svc.api handler reloads model from BentoML store on every request — 340ms load time drives p99 latency to 345ms vs 100ms SLA — engineer tunes max_batch_size and async patterns with no effect — BentoML developer on monthly retainer

BentoML 1.1.11 • LightGBM 4.1.0 • Python 3.11 • Kubernetes • @svc.api(batch=True) • max_batch_size=32 • max_latency_ms=100 • runner initialized inside handler • 340ms model load per request • p99 345ms vs 100ms SLA • 100 req/s • retainer billing

The credit scoring team at Meridian Credit Intelligence LLC had a BentoML service that worked perfectly in staging and fell apart in production. At 100 requests per second, the p99 latency was 345ms — 3.45× the 100ms SLA. The ML engineer spent two weeks investigating: tuning max_batch_size from 32 to 64 to 128, adjusting max_latency_ms, switching from synchronous to async/await runner calls, and adding horizontal pod replicas. None of it helped. The BentoML retainer developer was brought in on the fourth week. The root cause was a single misplaced line: the LightGBM runner was being initialized inside the @svc.api handler function, causing the model to be loaded from the BentoML store — 340ms — on every incoming request.

Meridian Credit Intelligence ran a real-time credit risk scoring service built on BentoML 1.1.11. The service wrapped a LightGBM 4.1.0 model trained on a 240,000-row credit dataset with 47 features. The service was deployed to Kubernetes with adaptive batching configured: max_batch_size=32, max_latency_ms=100. The SLA required p99 latency under 100ms at up to 200 requests per second. In staging, with one request per second from the integration test suite, the service responded in 350ms on the first request and 8ms on subsequent requests — the first-request overhead was noted but not investigated, because integration tests sent requests sequentially with multi-second gaps and never triggered the per-request reload pattern under sustained load.

# credit_scoring_service.py — Meridian Credit Intelligence LLC
# BentoML 1.1.11, LightGBM 4.1.0, Python 3.11, Kubernetes
# Deployed to production; p99 latency 345ms vs 100ms SLA at 100 req/s.

import bentoml
from bentoml.io import NumpyNdarray
import numpy as np

svc = bentoml.Service("credit_scoring_service")

@svc.api(
    input=NumpyNdarray(),
    output=NumpyNdarray(),
    batch=True
)
async def predict(input_array: np.ndarray) -> np.ndarray:
    # BUG: runner initialized INSIDE the API handler function.
    # Every incoming request (or batch) calls this function.
    # bentoml.lightgbm.get("credit_model:latest") reads model metadata
    # from the BentoML model store (NFS volume mount on Kubernetes).
    # .to_runner() initializes the runner, loading the LightGBM booster
    # from disk into memory.
    # Model load time: ~340ms (47-feature, 500-tree LightGBM model,
    #   serialized as 18MB .txt file on NFS).
    # This 340ms overhead applies to EVERY request at 100 req/s.
    runner = bentoml.lightgbm.get("credit_model:latest").to_runner()
    # Actual inference: ~5ms.
    result = await runner.async_run(input_array)
    return result

# At 100 req/s:
# Request latency = 340ms (model load) + 5ms (LightGBM inference) = 345ms p99
# SLA: 100ms p99
# Ratio: 345ms / 100ms = 3.45× over SLA
# max_batch_size=32 and max_latency_ms=100 do not affect model load overhead.
# async/await does not affect model load overhead (it's still synchronous I/O).
# Horizontal pod scaling reduces throughput pressure but not per-request latency.

The ML engineer’s first hypothesis was that the adaptive batching configuration was causing requests to queue past the 100ms timeout. She increased max_batch_size from 32 to 64, then to 128. Latency was unchanged at 345ms. Her second hypothesis was that synchronous runner execution was blocking the async event loop. She changed the runner call from synchronous to await runner.async_run(). Latency was unchanged. Her third hypothesis was resource contention — the Kubernetes pod was CPU-constrained, causing LightGBM inference to take longer than the staged measurement. She added two more pod replicas. Latency remained 345ms. After two weeks and four hypothesis-driven changes with no improvement, the team requested retainer support.

The BentoML retainer developer’s first step was a service profiling run — adding timing instrumentation to decompose the 345ms request latency into its components.

# Profiling instrumentation — BentoML retainer developer
# Added timing breakpoints to credit_scoring_service.py

import time

@svc.api(input=NumpyNdarray(), output=NumpyNdarray(), batch=True)
async def predict_instrumented(input_array: np.ndarray) -> np.ndarray:
    t0 = time.perf_counter()

    runner = bentoml.lightgbm.get("credit_model:latest").to_runner()
    t1 = time.perf_counter()
    # t1 - t0: 338–342ms — model load from BentoML store (NFS)

    result = await runner.async_run(input_array)
    t2 = time.perf_counter()
    # t2 - t1: 4–6ms — LightGBM inference (47 features, 500 trees)

    print(f"model_load={t1-t0:.3f}s inference={t2-t1:.3f}s total={t2-t0:.3f}s")
    return result

# Profiling output (100 sequential requests at 100 req/s):
# model_load=0.340s inference=0.005s total=0.345s
# model_load=0.339s inference=0.005s total=0.344s
# model_load=0.341s inference=0.005s total=0.346s
# ...
# 340ms is model loading, not inference. Root cause: runner in handler scope.

The profiling output made the root cause unambiguous: 98.5% of the total request latency was model loading, not LightGBM inference. The fix was moving the runner initialization from inside the handler function to module level — a two-line change. At module level, the runner is initialized once when the BentoML service process starts up. The model is loaded from the BentoML store into memory during startup, which takes 340ms. Every subsequent request uses the in-memory runner, paying only the 5ms inference cost.

# credit_scoring_service.py — FIXED
# BentoML 1.1.11, LightGBM 4.1.0, Python 3.11, Kubernetes

import bentoml
from bentoml.io import NumpyNdarray
import numpy as np

# FIXED: runner initialized at MODULE LEVEL.
# bentoml.lightgbm.get("credit_model:latest") executes once at service startup.
# Model is loaded from BentoML store (340ms, paid once).
# All API handler calls use the in-memory runner (5ms per request).
runner = bentoml.lightgbm.get("credit_model:latest").to_runner()

svc = bentoml.Service("credit_scoring_service", runners=[runner])

@svc.api(
    input=NumpyNdarray(),
    output=NumpyNdarray(),
    batch=True
)
async def predict(input_array: np.ndarray) -> np.ndarray:
    # No runner initialization here. Uses module-level runner.
    result = await runner.async_run(input_array)
    return result

# At 100 req/s after fix:
# Request latency = 0ms (model already in memory) + 5ms (inference) = ~5ms p99
# Startup latency = 340ms (model load, paid once during pod startup)
# SLA: 100ms p99 — met with 20× headroom.

After the fix was deployed to the Kubernetes cluster, p99 latency dropped from 345ms to 8ms at 100 requests per second. At 200 requests per second (the peak load target), p99 latency measured 11ms — well within the 100ms SLA. Service startup time increased by 340ms (the model loading time) but this was irrelevant for a Kubernetes deployment with rolling updates and readiness probes.

The retainer developer also added a service initialization check to the CI pipeline: a static analysis script that greps all BentoML service files for the pattern bentoml\.\w+\.get( inside function bodies (using the AST module to detect function-scoped assignments to .to_runner()) and fails the build if any are found. This prevents the per-request runner initialization pattern from being reintroduced in future model deployments.

Why per-request runner initialization is a recurring BentoML retainer pattern

The per-request runner initialization failure is common because it is invisible in development. A BentoML service tested with infrequent manual requests shows high latency only on the first request — subsequent requests in the same process reuse the runner and are fast. This latency pattern looks like a normal “cold start” effect, and engineers who observe it in development often don’t recognize it as a correctness issue. The service definition syntax doesn’t enforce where the runner is initialized — BentoML does not raise an error when a runner is initialized inside a handler function. The documentation recommends module-level initialization but does not prevent the in-handler pattern.

Under production load, the per-request runner initialization becomes the dominant cost in every request. Because the model load time is constant (340ms, regardless of input size or batch size), adaptive batching configuration has no effect: increasing max_batch_size amortizes the inference compute cost across more inputs but not the constant model load cost. Async execution has no effect: the model load is synchronous file I/O that blocks the coroutine regardless of whether the outer call is await runner.async_run(). Horizontal scaling has no effect on per-request latency: each replica still loads the model on every request. Only moving the runner initialization to module level eliminates the per-request model load overhead.

The 4.0-hour engagement covered investigation (1.5h — 0.5h latency profiling: 340ms model load + 5ms inference = 345ms total; 0.5h runner initialization scope analysis: runner = bentoml.lightgbm.get(...).to_runner() inside @svc.api handler confirmed; 0.5h hypothesis elimination: adaptive batching, async execution, and horizontal scaling each tested and ruled out as root cause), fix implementation (1.0h — 0.5h module-level runner initialization deployed to staging; 0.5h load test: p99 8ms at 100 req/s, 11ms at 200 req/s; SLA met), and validation + prevention (1.5h — 0.5h production deployment with rolling update; 0.5h 24-hour latency monitoring: p99 stable at 8–11ms; 0.5h static analysis check added to CI pipeline to detect runner initialization inside function bodies).

In a HourTab retainer log, the incident session reads: “BentoML per-request runner initialization investigation: BentoML 1.1.11, LightGBM 4.1.0, Python 3.11, Kubernetes; credit_scoring_service; runner = bentoml.lightgbm.get('credit_model:latest').to_runner() inside @svc.api handler; model loaded from BentoML store (NFS) on every request; model load time 340ms (47-feature, 500-tree LightGBM, 18MB .txt); LightGBM inference ~5ms; p99 latency 345ms at 100 req/s vs 100ms SLA; 2 weeks of tuning max_batch_size, async patterns, horizontal scaling — no effect; fix: runner initialization moved to module level; model loads once at service startup; p99 8ms at 100 req/s; 4.0h.”

HourTab turns a time-tracker CSV into a public retainer-hours URL your client can bookmark — no login, no portal, no “how many hours do I have left?” emails. Free for one retainer. $9/mo for up to ten.

FAQ: BentoML developer retainers

See also: Kubeflow developer retainer · LightGBM developer retainer · MLflow developer retainer