Seldon Core developer on retainer: SeldonDeployment predictor image updated for retrained model but transformer image (StandardScaler) not updated — new XGBoost model receives features normalized to old dataset distribution — production AUC drops 0.91→0.79 in 2 hours — Seldon Core developer on monthly retainer

Seldon Core 1.17.1 • Scikit-learn 1.3.2 • XGBoost 2.0.0 • GKE 1.27 • SeldonDeployment transformer + predictor pipeline • StandardScaler v1.3 on v2.1 model • 180k→320k training rows • feature distribution shift • AUC 0.91→0.79 • 48-hour detection lag • retainer billing

On October 14, 2026, two days after what had appeared to be a routine model deployment, an ML engineer at Vantage Risk Analytics Corp. noticed the production AUC on the Grafana dashboard had dropped from 0.91 to 0.79. The deployment had succeeded cleanly: all pods were Running, no errors in the logs, requests processing normally. The Seldon Core retainer developer was engaged on October 15 and traced the root cause in two hours: the SeldonDeployment’s predictor image had been updated to deploy the newly retrained XGBoost model, but the transformer image — containing the StandardScaler fitted on the original training data — had been left at the old version. The new model had been trained on an expanded dataset with different feature distributions. The old scaler was normalizing inference inputs using the wrong means and standard deviations, and the new model was receiving features that didn’t match its training distribution.

Vantage Risk Analytics ran a credit risk scoring pipeline deployed as a Seldon Core SeldonDeployment on GKE. The pipeline had two components: a transformer (a Python container with a sklearn.preprocessing.StandardScaler fitted on the training data, loaded at startup from scaler.joblib) and a predictor (an XGBoost model served via Seldon’s MLServer with the mlserver-xgboost runtime). The transformer preprocessed raw inference inputs and passed normalized features to the predictor. The original model had been trained on 180,000 rows of credit application data covering three domestic markets, with 52 features. The scaler was fitted on this 180,000-row dataset, and the transformer Docker image (credit-transformer:v1.3) contained the fitted scaler serialized as scaler.joblib.

# credit_risk_pipeline.yaml — Vantage Risk Analytics Corp.
# Seldon Core 1.17.1 on GKE 1.27
# Production SeldonDeployment after October 12 2026 update.

apiVersion: machinelearning.seldon.io/v1
kind: SeldonDeployment
metadata:
  name: credit-risk-pipeline
  namespace: ml-serving
spec:
  predictors:
  - name: default
    graph:
      name: transformer
      type: TRANSFORMER
      endpoint:
        type: REST
      children:
      - name: predictor
        type: MODEL
        endpoint:
          type: REST
    componentSpecs:
    - spec:
        containers:
        - name: transformer
          # BUG: transformer image NOT updated to match new model.
          # credit-transformer:v1.3 contains StandardScaler fitted on
          # 180,000-row original dataset (3 markets, 52 features).
          # New XGBoost model (credit-predictor:v2.1) was trained on
          # 320,000-row expanded dataset (6 markets, 67 features).
          # Feature distributions differ significantly between v1 and v2 training data.
          # Old scaler's means/stds don't match new model's training distribution.
          image: gcr.io/vantage-risk/credit-transformer:v1.3  # BUG: should be v2.1
          resources:
            requests:
              memory: "512Mi"
              cpu: "250m"
        - name: predictor
          # Predictor correctly updated to new XGBoost model.
          image: gcr.io/vantage-risk/credit-predictor:v2.1   # Correctly updated
          resources:
            requests:
              memory: "2Gi"
              cpu: "500m"

In September 2026, Vantage Risk Analytics expanded the training dataset for the next model version. The data science team added three new markets (six markets total), extended the historical window by 18 months, and incorporated 15 new features — growing the dataset from 180,000 to 320,000 rows and from 52 to 67 features. The new dataset had noticeably different feature distributions from the original: four features had mean shifts greater than 0.5 standard deviations, and eleven features had standard deviation ratios greater than 1.3 between the old and new training data. The data science team refitted the StandardScaler on the new 320,000-row dataset — as expected, since the scaler must always be fitted on the same data as the model — and trained a new XGBoost model. The new model achieved AUC 0.933 in cross-validation and 0.927 on the holdout test set.

# training_pipeline.py — Vantage Risk Analytics (simplified)
# September 2026 retraining run

from sklearn.preprocessing import StandardScaler
from sklearn.model_selection import train_test_split
import xgboost as xgb
import joblib

# New dataset: 320,000 rows, 67 features (3 new markets + 15 new features)
X_train, X_test, y_train, y_test = train_test_split(X_new, y_new, ...)

# Scaler correctly fitted on NEW training data
scaler_v2 = StandardScaler()
X_train_scaled = scaler_v2.fit_transform(X_train)
X_test_scaled = scaler_v2.transform(X_test)

# scaler_v2 parameters differ from scaler_v1:
# feature_0: mean 0.42→0.61 (+0.19), std 1.12→1.38 (+23%)
# feature_7: mean 2.14→2.89 (+0.75), std 0.88→1.21 (+38%)
# feature_23: mean -0.31→0.08 (+0.39), std 1.44→1.19 (-17%)
# ... (11 features with std ratio >1.3, 4 features with mean shift >0.5 std)

model_v2 = xgb.XGBClassifier(n_estimators=500, max_depth=6, learning_rate=0.05)
model_v2.fit(X_train_scaled, y_train)
# Test AUC: 0.927

# scaler_v2 saved for transformer image build
joblib.dump(scaler_v2, "artifacts/scaler_v2.joblib")  # Correct: new scaler
# model_v2 saved for predictor image build
model_v2.save_model("artifacts/credit_model_v2.json")

# The deployment engineer builds credit-predictor:v2.1 with credit_model_v2.json.
# But credit-transformer:v2.1 with scaler_v2.joblib is NOT built — overlooked.
# Only credit-predictor:v2.1 is deployed to production SeldonDeployment.
# credit-transformer:v1.3 (old scaler, old data) remains in production.

The deployment happened on October 12, 2026. The MLOps engineer updated the SeldonDeployment YAML: credit-predictor:v2.1 for the predictor component, credit-transformer:v1.3 unchanged. The SeldonDeployment rollout succeeded. The transformer pod and predictor pod restarted with the new configuration. The Kubernetes readiness probes passed. There were no errors in the pod logs.

In production, every inference request was preprocessed by the credit-transformer:v1.3 container, which loaded scaler_v1.joblib at startup — the scaler fitted on the 180,000-row original dataset. The transformer normalized inference inputs using the old feature means and standard deviations and passed the normalized features to credit-predictor:v2.1. The new XGBoost model had learned its decision boundaries from features normalized with scaler_v2 (new data distributions). Receiving features normalized with scaler_v1 (old data distributions) put inference inputs in a systematically different region of the feature space than the model had encountered in training. The production AUC dropped from 0.91 to 0.79 immediately after the deployment — a 12 percentage point regression.

# Investigation — Seldon Core retainer developer, October 15 2026
# Vantage Risk Analytics Corp., Seldon Core 1.17.1, GKE 1.27

# Step 1: Verify SeldonDeployment component image tags
kubectl get seldondeployment credit-risk-pipeline -n ml-serving -o yaml | grep "image:"
# transformer: gcr.io/vantage-risk/credit-transformer:v1.3  ← OLD
# predictor:   gcr.io/vantage-risk/credit-predictor:v2.1   ← NEW
# Mismatch confirmed.

# Step 2: Check scaler artifact dates in each image
docker run --rm gcr.io/vantage-risk/credit-transformer:v1.3 \
    python -c "import joblib; s = joblib.load('/app/scaler.joblib'); print(s.mean_[:5])"
# [0.42, 1.87, -0.15, 3.21, 0.08] — scaler_v1, old distribution

docker run --rm gcr.io/vantage-risk/credit-transformer:v2.1 \
    python -c "import joblib; s = joblib.load('/app/scaler.joblib'); print(s.mean_[:5])"
# [0.61, 2.04, -0.08, 3.45, 0.22] — scaler_v2, new distribution (credit-transformer:v2.1 exists)
# The v2.1 transformer image was built but never deployed.

# Step 3: Confirm AUC impact of v1.3 scaler on v2.1 model
# Offline simulation: pass v2 test data through v1 scaler → v2 model
X_test_v1_scaled = scaler_v1.transform(X_test_new)  # Old scaler on new test data
auc_with_v1_scaler = roc_auc_score(y_test_new, model_v2.predict_proba(X_test_v1_scaled)[:,1])
# auc_with_v1_scaler: 0.791 — matches observed production AUC drop

X_test_v2_scaled = scaler_v2.transform(X_test_new)  # Correct: new scaler on new test data
auc_with_v2_scaler = roc_auc_score(y_test_new, model_v2.predict_proba(X_test_v2_scaled)[:,1])
# auc_with_v2_scaler: 0.927 — expected production AUC after fix

The fix was updating the transformer image tag in the SeldonDeployment manifest from credit-transformer:v1.3 to credit-transformer:v2.1. The v2.1 transformer image had been built correctly during the September training run — it was sitting in the container registry, unused, because the deployment step had been executed incompletely. After the SeldonDeployment was updated and the transformer pod restarted with v2.1, production AUC recovered to 0.912 within four hours — consistent with the offline test AUC of 0.927, the small gap attributable to production data distribution shift over the preceding month.

# credit_risk_pipeline.yaml — FIXED
# October 16 2026: transformer image updated to v2.1

apiVersion: machinelearning.seldon.io/v1
kind: SeldonDeployment
metadata:
  name: credit-risk-pipeline
  namespace: ml-serving
spec:
  predictors:
  - name: default
    graph:
      name: transformer
      type: TRANSFORMER
      children:
      - name: predictor
        type: MODEL
    componentSpecs:
    - spec:
        containers:
        - name: transformer
          # FIXED: transformer updated to v2.1 (scaler fitted on 320k-row expanded dataset)
          image: gcr.io/vantage-risk/credit-transformer:v2.1
          resources:
            requests:
              memory: "512Mi"
              cpu: "250m"
        - name: predictor
          image: gcr.io/vantage-risk/credit-predictor:v2.1
          resources:
            requests:
              memory: "2Gi"
              cpu: "500m"

# Production AUC after fix (October 16 2026):
# 4 hours post-deployment: AUC 0.912
# 24 hours post-deployment: AUC 0.909 (stable)
# Pre-regression baseline: AUC 0.910 (consistent)

The retainer developer also implemented a prevention measure: a deployment validation script added to the CI/CD pipeline that reads the SeldonDeployment manifest before any deployment and verifies that all component image tags share the same version suffix when the pipeline topology includes a transformer-predictor pair. The script fails the deployment pipeline if a transformer image version doesn’t match the predictor image version for the same model lineage, requiring an explicit override flag with a justification comment to proceed with mismatched versions.

Why transformer-predictor image version mismatch is a recurring Seldon retainer pattern

The transformer-predictor version mismatch failure occurs because Seldon Core’s component graph architecture separates preprocessing from prediction into independently versioned Docker images. This separation is architecturally sound — it allows transformers and predictors to be scaled, updated, and debugged independently. But it introduces a coupling requirement that is easy to violate: whenever a model is retrained on new data that shifts the feature distribution, the transformer must be updated to reflect the new feature statistics. This coupling isn’t enforced by Kubernetes or by Seldon Core — the SeldonDeployment YAML accepts any valid image tag for each component.

The failure is hard to detect immediately because it doesn’t produce errors. The transformer normalizes inputs correctly (from its perspective — it’s applying scaler_v1.transform() faithfully). The predictor makes predictions correctly (from its perspective — it’s applying the XGBoost model to the features it receives). The end-to-end system is internally consistent — it’s just producing wrong predictions. The only signal is model performance metrics. If AUC monitoring has an alert threshold set at 3+ percentage point regression (a common conservative threshold to avoid false positives from normal daily variation), a 12-point regression will trigger the alert — but 48 hours of data may have already been scored with the wrong model behavior before the alert fires and is investigated.

A Seldon Core retainer developer who implements a post-deployment AUC validation window catches this failure class in hours rather than days. The standard practice: after every SeldonDeployment update, monitor production AUC for 4 hours against the pre-deployment baseline. Any regression greater than 2 percentage points triggers an immediate rollback, regardless of whether the deployment logs show errors. The 4-hour window is short enough to limit business impact while long enough to accumulate sufficient inference volume for statistically reliable AUC estimation.

The 5.5-hour engagement covered investigation (2.0h — 0.5h SeldonDeployment YAML review: transformer v1.3, predictor v2.1 mismatch; 0.5h scaler artifact inspection: scaler_v1.mean_ vs scaler_v2.mean_ confirmed distribution shift; 0.5h offline simulation: v2 test data through v1 scaler → v2 model → AUC 0.791, matches production; 0.5h container registry audit: credit-transformer:v2.1 exists in registry, was never deployed), fix implementation (1.5h — 0.5h SeldonDeployment YAML updated: transformer image to v2.1; 0.5h rolling update applied with readiness probe verification; 0.5h AUC monitoring: 0.912 within 4 hours, 0.909 stable at 24 hours), and prevention (2.0h — 1.0h deployment validation script: checks transformer-predictor image version suffix alignment; 1.0h deployment checklist documented: transformer and predictor images must share the same version for any retrain deployment; post-deployment AUC monitoring SLA: 4-hour window, 2% regression threshold).

In a HourTab retainer log, the incident session reads: “Seldon Core transformer-predictor image version mismatch investigation: Seldon Core 1.17.1, Sklearn 1.3.2, XGBoost 2.0.0, GKE 1.27; credit-risk-pipeline SeldonDeployment; transformer → predictor graph topology; September 2026 model retrained on expanded dataset (180k→320k rows, 52→67 features, 3 new markets); new StandardScaler fitted on new data (mean/std shifts on 11 features); predictor image updated to v2.1; transformer image left at v1.3; inference inputs normalized with v1 scaler parameters → v2 model receives features outside training distribution; AUC 0.91→0.79 within 2 hours of deployment; discovered October 14 (48h later); credit-transformer:v2.1 existed in registry but was never deployed; fix: transformer image updated to v2.1; AUC recovers to 0.912; 5.5h.”

HourTab turns a time-tracker CSV into a public retainer-hours URL your client can bookmark — no login, no portal, no “how many hours do I have left?” emails. Free for one retainer. $9/mo for up to ten.

FAQ: Seldon Core developer retainers

See also: Kubeflow developer retainer · BentoML developer retainer · MLflow developer retainer