Ray Serve developer on retainer: StandardScaler fitted on per-replica warm-up request batch in __init__ — 4 replicas have different feature means — same feature value produces different predictions depending on replica assignment — P99 AUC varies 8 percentage points — Ray Serve developer on monthly retainer

Ray 2.9.1 • Ray Serve 2.9.1 • scikit-learn 1.3.2 • GKE 1.28 • CascadeChurnDeployment • num_replicas=4 • StandardScaler fitted on first 200 requests in __init__ • replica 0 feature_7 mean=0.42, replica 3 mean=1.91 • 8pt AUC variance • A/B test corrupted • 4.5h retainer engagement

During an A/B test in October 2026, the ML team at Cascade Inference Systems noticed that the “control” cohort of their churn prediction model showed unexpectedly high variance in model scores for users with similar feature profiles. The variance exceeded statistical noise by six standard deviations, which suggested a systematic data partitioning or model serving problem. The Ray Serve retainer developer was engaged and identified the cause within two hours: the CascadeChurnDeployment class was fitting a StandardScaler on the first 200 inference requests received by each replica in its __init__ method. With num_replicas=4, each of the four replicas received a different slice of the warm-up traffic during startup. Round-robin load balancing then randomly assigned subsequent requests to any replica. The same feature value produced different normalized values — and different model predictions — depending on which replica handled the request.

Cascade Inference Systems deployed a scikit-learn churn prediction model using Ray Serve 2.9.1. The deployment class loaded the model from S3 in __init__, which was correct. The developer had also added a StandardScaler to normalize the inference features to match the distribution of production traffic, reasoning that the training data was older and the feature distributions might have drifted. The scaler was fitted on the first 200 requests received by each replica — a lazy warm-up approach intended to ensure the normalization reflected recent production values. The developer tested this with a single replica (num_replicas=1) on staging, where it worked correctly. When the deployment was scaled to num_replicas=4 in production, the problem appeared immediately but was not detected because no per-replica monitoring was in place.

# serving/deployment.py — Cascade Inference Systems
# Ray 2.9.1, Ray Serve 2.9.1, scikit-learn 1.3.2, GKE 1.28

import ray
from ray import serve
import boto3, joblib, io
import numpy as np
from sklearn.preprocessing import StandardScaler

@serve.deployment(
    num_replicas=4,
    ray_actor_options={"num_cpus": 2, "num_gpus": 0},
    max_concurrent_queries=20
)
class CascadeChurnDeployment:
    def __init__(self):
        # Load the scikit-learn churn model from S3 — correct.
        s3 = boto3.client('s3')
        model_bytes = s3.get_object(Bucket='cascade-models', Key='churn/v23/model.pkl')['Body'].read()
        self.model = joblib.load(io.BytesIO(model_bytes))

        # BUG: Fit a new StandardScaler on the first 200 inference requests received by THIS replica.
        # With num_replicas=4, each replica runs __init__ independently.
        # Each replica will fit its scaler on a different slice of warm-up traffic.
        # Round-robin routing means different replicas see different user cohorts.
        # Replica 0 warm-up: early-morning users (low activity cohort), feature_7 mean=0.42.
        # Replica 1 warm-up: mid-morning users, feature_7 mean=0.89.
        # Replica 2 warm-up: late-morning users, feature_7 mean=1.43.
        # Replica 3 warm-up: mid-day users (high activity cohort), feature_7 mean=1.91.
        # Training data feature_7 mean=1.17 — none of the replicas match.
        self.scaler = StandardScaler()
        self._warmup_buffer = []
        self._warmed_up = False

    async def __call__(self, request):
        features = np.array(await request.json())

        if not self._warmed_up:
            self._warmup_buffer.append(features)
            if len(self._warmup_buffer) >= 200:
                # Fit scaler on this replica's warm-up batch.
                self.scaler.fit(np.vstack(self._warmup_buffer))
                self._warmed_up = True
                self._warmup_buffer = []  # Free memory

        if self._warmed_up:
            features = self.scaler.transform(features.reshape(1, -1))
        # During warm-up (first 200 requests), no normalization — also incorrect behavior.

        score = self.model.predict_proba(features)[0][1]
        return {"churn_score": float(score)}

The problem became visible during an A/B test comparing two versions of a downstream recommendation algorithm. The control cohort’s churn scores showed high variance across users with similar feature profiles. Users with feature_7 = 1.5 (the median value in the test population) received churn scores between 0.61 and 0.73, depending on which Ray Serve replica processed their request. The 0.12 absolute score difference (8 percentage points of AUC) was large enough to affect downstream business decisions — users above 0.65 were flagged for retention outreach, so a single user could receive different retention actions on different API calls.

The retainer developer diagnosed the issue by instrumenting replica routing. Ray Serve’s serve.get_replica_context() API exposes the replica ID within the deployment handler. The developer added a response header with the replica ID and collected 10,000 requests, then segmented the feature_7 distribution by replica:

# Diagnosis: instrument replica routing — added October 22, 2026
# Temporary diagnostic version — not the production fix.

from ray import serve
import numpy as np

@serve.deployment(num_replicas=4, ...)
class CascadeChurnDeploymentDiagnostic:
    async def __call__(self, request):
        features = np.array(await request.json())
        replica_context = serve.get_replica_context()
        replica_id = replica_context.replica_id

        if self._warmed_up:
            features_normalized = self.scaler.transform(features.reshape(1, -1))
            # Log scaler mean for feature_7 alongside prediction for comparison
            scaler_mean_f7 = float(self.scaler.mean_[7])
        else:
            features_normalized = features
            scaler_mean_f7 = None

        score = self.model.predict_proba(features_normalized.reshape(1, -1))[0][1]

        # Return replica ID and scaler parameters in response for analysis
        return {
            "churn_score": float(score),
            "replica_id": str(replica_id),
            "scaler_mean_f7": scaler_mean_f7
        }

# Analysis after 10,000 requests:
# replica_0: scaler_mean_f7=0.42, mean churn_score=0.71 for feature_7=1.5
# replica_1: scaler_mean_f7=0.89, mean churn_score=0.68 for feature_7=1.5
# replica_2: scaler_mean_f7=1.43, mean churn_score=0.64 for feature_7=1.5
# replica_3: scaler_mean_f7=1.91, mean churn_score=0.61 for feature_7=1.5
# Training scaler mean_f7=1.17 → expected score=0.66 for feature_7=1.5
# Range: 0.61–0.73 (0.12 absolute, ~8 percentage points AUC)

The fix was straightforward: load the pre-fitted StandardScaler artifact from the same S3 path as the model, removing all request-time fitting logic from __init__. The training pipeline was already serializing the scaler alongside the model, so no changes to the training pipeline were required. The deployment was updated to load the frozen scaler artifact:

# serving/deployment.py — fixed October 22, 2026

import ray
from ray import serve
import boto3, joblib, io
import numpy as np

@serve.deployment(
    num_replicas=4,
    ray_actor_options={"num_cpus": 2, "num_gpus": 0},
    max_concurrent_queries=20
)
class CascadeChurnDeployment:
    def __init__(self):
        s3 = boto3.client('s3')

        # Load the scikit-learn churn model — unchanged.
        model_bytes = s3.get_object(
            Bucket='cascade-models', Key='churn/v23/model.pkl'
        )['Body'].read()
        self.model = joblib.load(io.BytesIO(model_bytes))

        # FIXED: Load the pre-fitted StandardScaler artifact from S3.
        # The scaler was fitted on the full training dataset during model training
        # and serialized to S3 alongside the model.
        # All 4 replicas load the SAME scaler artifact → byte-for-byte identical parameters.
        # feature_7 mean = 1.17 (training dataset mean) on all replicas.
        scaler_bytes = s3.get_object(
            Bucket='cascade-models', Key='churn/v23/scaler.pkl'
        )['Body'].read()
        self.scaler = joblib.load(io.BytesIO(scaler_bytes))
        # No warm-up buffer. No lazy fitting. Scaler is ready immediately.

    async def __call__(self, request):
        features = np.array(await request.json())
        features_normalized = self.scaler.transform(features.reshape(1, -1))
        score = self.model.predict_proba(features_normalized)[0][1]
        return {"churn_score": float(score)}

After the fix was deployed to all four replicas, the retainer developer ran a validation test with 50,000 requests across all replicas and confirmed that the P99 AUC variance across replicas was within 0.3 percentage points — within the expected noise floor for a balanced random routing policy. The A/B test was restarted from scratch on October 23, 2026, with clean baselines.

The retainer developer also added a quarterly preprocessing audit task to the Ray Serve runbook: before each model update deployment, a CI job inspects the deployment class source for any instantiation of StandardScaler, MinMaxScaler, LabelEncoder, TfidfVectorizer, or other stateful preprocessing classes that are not immediately followed by a .load() call. If a stateful object is instantiated without being loaded from a frozen artifact, the CI job fails and the deployment is blocked until the issue is resolved. This prevents future regressions of the per-replica divergence pattern.

Total retainer engagement: 4.5 hours (0.5h A/B test variance alert: 6-sigma control cohort noise; 1.5h instrument replica routing with serve.get_replica_context() and collect 10,000 tagged predictions; 1.0h confirm per-replica scaler mean divergence: feature_7 means 0.42/0.89/1.43/1.91 across 4 replicas; 0.5h fix: update __init__ to load pre-fitted scaler from S3; 0.5h validation: 50k requests, P99 AUC variance within 0.3pt; 0.5h prevention: CI preprocessing audit + quarterly scaler-model pairing review). The retainer was billed in 0.5h increments, logged to HourTab with the Ray version, deployment class name, and replica scaler means before and after the fix. The client verified the complete 4.5-hour work log at the shared HourTab URL without logging in.

Related retainer posts

• BentoML developer on retainer — model serving and deployment retainer work
• Seldon developer on retainer — Seldon Core model serving retainer work
• Kubeflow developer on retainer — ML pipeline platform retainer work