scikit-learn developer on retainer: Pipeline StandardScaler transforms columns by position not name — inference DataFrame column order differs from training — wrong feature scaled against wrong statistics — 3 weeks — scikit-learn developer on monthly retainer

scikit-learn 1.4.x • Python 3.11 • joblib 1.3.x • Pipeline([StandardScaler, LogisticRegression]) • 50,000-row training set • 4 numeric features • training columns: ['income', 'age', 'credit_score', 'years_employed'] • inference columns from new CSV: ['age', 'credit_score', 'income', 'years_employed'] • StandardScaler.transform() applies statistics by position not name • age standardized with income statistics • predictions plausible but wrong • AUC-ROC drops from 0.89 to 0.61 at inference • developer checks feature values, encoding, model — 3 weeks • fix: reorder inference columns with feature_names_in_ • add column assertion • 1.5h retainer engagement

Northfield Analytics Ltd. built their loan approval model on scikit-learn in September 2026, training a Pipeline with a StandardScaler preprocessing step and a LogisticRegression classifier on 50,000 applications. The training data came from their internal CRM export, where columns were specified manually in a domain-meaningful order: ['income', 'age', 'credit_score', 'years_employed']. The pipeline achieved an AUC-ROC of 0.89 on held-out validation data. The team serialized the fitted pipeline with joblib.dump() and deployed it to an inference service that scored new applications in real time.

Three months later, Northfield migrated their data ingestion pipeline to a new vendor whose API returned applicant features alphabetically sorted: ['age', 'credit_score', 'income', 'years_employed']. The inference service loaded the new data directly into a DataFrame and passed it to the pipeline. No error was raised. But the AUC-ROC of the deployed model, measured on a holdout set of newly scored applications, had dropped to 0.61. The model was approving loans it should decline and declining loans it should approve. The developer investigated feature values, data encoding, and LogisticRegression coefficients for three weeks. They engaged a scikit-learn retainer developer.

The inference pipeline at the time of the engagement:

# inference.py
# Northfield Analytics Ltd.
# scikit-learn 1.4.x, Python 3.11, joblib 1.3.x

import joblib
import pandas as pd

# Load the fitted pipeline
pipeline = joblib.load("loan_pipeline.pkl")

def score_application(application: dict) -> float:
    """
    application: dict with keys 'age', 'credit_score',
                               'income', 'years_employed'
    Returns probability of approval (0.0 to 1.0)
    """
    # New vendor API returns fields alphabetically sorted
    df = pd.DataFrame([{
        "age":            application["age"],
        "credit_score":   application["credit_score"],
        "income":         application["income"],
        "years_employed": application["years_employed"],
    }])
    # df.columns: ['age', 'credit_score', 'income', 'years_employed']
    # Pipeline was trained on columns in order:
    #   ['income', 'age', 'credit_score', 'years_employed']
    #
    # StandardScaler.transform() processes column 0 with mean_[0] / scale_[0]
    # mean_[0] = mean of 'income' at training (e.g., 68,000)
    # But column 0 at inference is 'age' (e.g., 34)
    # 'age' 34 standardized with income statistics:
    #   (34 - 68000) / 42000 ≈ -1.618  ← absurd scaled value
    # 'income' 82000 standardized with age statistics:
    #   (82000 - 41.2) / 12.3 ≈ 6,664  ← absurd scaled value
    # No exception raised; predictions are returned

    prob = pipeline.predict_proba(df)[0][1]
    return prob

Why scikit-learn StandardScaler transforms by position, not by name

scikit-learn's StandardScaler (and all sklearn transformers) stores statistics during fit() as numpy arrays indexed by column position. StandardScaler.mean_ is a 1-D array where mean_[0] is the mean of the first column of the training data, mean_[1] is the mean of the second column, and so on. There is no internal mapping between column positions and column names stored in the scaler itself.

When transform() is called, it applies mean_[i] and scale_[i] to column i of the input array — regardless of what that column is named. The column name in a pandas DataFrame does not influence which statistics are applied.

scikit-learn 1.0 introduced feature_names_in_ on transformers fitted on DataFrames with string column names. In scikit-learn 1.1+, passing a DataFrame with different column names or in a different order raises a ValueError with a helpful message. But this check only fires when a DataFrame with named columns is passed directly to the transformer. In a Pipeline, prior steps may convert the DataFrame to a numpy array before it reaches the scaler, bypassing the name check entirely. In scikit-learn 1.4.x, a Pipeline fitted on a DataFrame does propagate feature name checks to each step — but only if each step is compatible with the set_output(transform="pandas") API. In a legacy pipeline where any intermediate step converts to numpy, the mismatch is invisible.

# Diagnosing the column-order mismatch — scikit-learn Pipeline inspection

import joblib
import numpy as np
import pandas as pd

pipeline = joblib.load("loan_pipeline.pkl")
scaler = pipeline.named_steps["scaler"]

# Step 1: inspect what feature names the scaler was fitted on
print("feature_names_in_:", scaler.feature_names_in_)
# feature_names_in_: ['income' 'age' 'credit_score' 'years_employed']
#                      ^^^^^^^^ ← training column order

# Step 2: inspect the scaler statistics
print("mean_:", scaler.mean_)
# mean_: [68412.3  41.2  682.5  7.4]
# position 0 = income mean (68,412), position 1 = age mean (41.2)

# Step 3: manually transform a known application with training column order
df_correct = pd.DataFrame([{
    "income": 82000, "age": 34,
    "credit_score": 720, "years_employed": 5,
}])
scaled_correct = scaler.transform(df_correct)
print("Correct order scaled:", scaled_correct)
# Correct order scaled: [[ 0.324  -0.585   0.981   -0.189]]
# income (82000 - 68412) / 42000 ≈ 0.32 — sensible

# Step 4: transform with inference column order (alphabetical — the bug)
df_wrong = pd.DataFrame([{
    "age": 34, "credit_score": 720,
    "income": 82000, "years_employed": 5,
}])
# In a legacy Pipeline, this may bypass the feature name check
# (depends on whether prior steps convert to numpy)
scaled_wrong = scaler.transform(df_wrong.values)  # pass numpy to bypass check
print("Wrong order scaled:", scaled_wrong)
# Wrong order scaled: [[-1.618e+00   5.67e+01   6.66e+03  -2.19e-01]]
# age (34) scaled with income statistics: (34 - 68412) / 21260 ≈ -1.62
# income (82000) scaled with age statistics: (82000 - 41.2) / 12.3 ≈ 6,664
# Absurd values — but passed to LogisticRegression without error

The fix: enforce column ordering at inference using feature_names_in_

The safe fix has two parts: reorder the inference DataFrame columns to match the training order using the pipeline's feature_names_in_ attribute, and add an explicit assertion that prevents mismatches from reaching the scaler. For long-term robustness, use ColumnTransformer in the pipeline with explicit column name selectors so that column ordering is enforced by the transformer itself rather than relying on the caller to provide the right order.

# inference.py — FIXED
# scikit-learn 1.4.x, Python 3.11, joblib 1.3.x

import joblib
import pandas as pd

pipeline = joblib.load("loan_pipeline.pkl")

# Retrieve the authoritative column order from the fitted pipeline
EXPECTED_FEATURES = list(pipeline.feature_names_in_)
# ['income', 'age', 'credit_score', 'years_employed']

def assert_feature_order(df: pd.DataFrame) -> pd.DataFrame:
    """
    Assert that df contains all expected features and return
    a copy with columns in the order the pipeline was trained on.
    Raises ValueError on missing or extra columns.
    """
    missing = set(EXPECTED_FEATURES) - set(df.columns)
    extra = set(df.columns) - set(EXPECTED_FEATURES)
    if missing:
        raise ValueError(
            f"Missing features: {missing}. "
            f"Expected: {EXPECTED_FEATURES}"
        )
    if extra:
        raise ValueError(
            f"Unexpected features: {extra}. "
            f"Expected: {EXPECTED_FEATURES}"
        )
    return df[EXPECTED_FEATURES]

def score_application(application: dict) -> float:
    """
    Returns probability of approval (0.0 to 1.0).
    Column order is enforced by assert_feature_order.
    """
    df = pd.DataFrame([application])
    df = assert_feature_order(df)
    # df.columns: ['income', 'age', 'credit_score', 'years_employed']
    # Matches training order — scaler applies correct statistics to each feature
    prob = pipeline.predict_proba(df)[0][1]
    return prob

Long-term fix: use ColumnTransformer with explicit selectors

The assert_feature_order guard above prevents the mismatch from causing silent errors. But the underlying fragility — that column ordering is an implicit contract not enforced by the pipeline itself — remains. The more robust pattern is to retrain the pipeline using ColumnTransformer, which selects columns by name rather than by position, so that even if the caller provides columns in the wrong order, the transformer applies the right statistics to the right column.

# training.py — ROBUST VERSION using ColumnTransformer
# scikit-learn 1.4.x, Python 3.11

from sklearn.compose import ColumnTransformer
from sklearn.preprocessing import StandardScaler
from sklearn.linear_model import LogisticRegression
from sklearn.pipeline import Pipeline
import pandas as pd
import joblib

# Explicitly name the features — order does not matter for inference
NUMERIC_FEATURES = ["income", "age", "credit_score", "years_employed"]

preprocessor = ColumnTransformer(
    transformers=[
        ("scaler", StandardScaler(), NUMERIC_FEATURES),
    ],
    remainder="drop",
)

pipeline = Pipeline([
    ("preprocessor", preprocessor),
    ("clf", LogisticRegression(max_iter=1000)),
])

# Train on 50,000 applications
X_train = pd.read_csv("applications_train.csv")
y_train = X_train.pop("approved")
pipeline.fit(X_train, y_train)

# ColumnTransformer selects columns by name, not position.
# Inference can provide columns in any order — the transformer
# will select and scale the correct column for each statistic.

joblib.dump(pipeline, "loan_pipeline_v2.pkl")

# Inference with ColumnTransformer — column order irrelevant
pipeline2 = joblib.load("loan_pipeline_v2.pkl")

# These two calls produce identical results regardless of column order:
df_a = pd.DataFrame([{"income": 82000, "age": 34, "credit_score": 720, "years_employed": 5}])
df_b = pd.DataFrame([{"age": 34, "credit_score": 720, "income": 82000, "years_employed": 5}])

assert pipeline2.predict_proba(df_a)[0][1] == pipeline2.predict_proba(df_b)[0][1]
# ColumnTransformer selects columns by name — same result regardless of order

Validation

# Validation log — November 2026
# scikit-learn 1.4.x, Python 3.11, joblib 1.3.x

# ---- Before fix ----
# Training column order: ['income', 'age', 'credit_score', 'years_employed']
# Inference column order (new vendor): ['age', 'credit_score', 'income', 'years_employed']
# StandardScaler applies income statistics (mean=68412) to age column (values 25-65)
# StandardScaler applies age statistics (mean=41.2) to income column (values 20k-200k)
# Inference AUC-ROC: 0.61 (vs 0.89 training evaluation)
# No errors raised in logs

# ---- After fix: reorder columns with feature_names_in_ ----
# pipeline.feature_names_in_: ['income' 'age' 'credit_score' 'years_employed']
# assert_feature_order() reorders inference DataFrame to match training order
# StandardScaler now applies correct statistics to each feature

# ---- Validation results ----
# Inference AUC-ROC: 0.89 (matches training evaluation)          ✓
# assert_feature_order() test with alphabetical order: passes     ✓
# assert_feature_order() test with missing column: raises ValueError ✓
# assert_feature_order() test with extra column: raises ValueError  ✓

# ---- Hour breakdown ----
# Triage:     0.5h — inference AUC-ROC 0.61 vs training 0.89;
#             no errors in logs; feature values correct in raw data
# Root cause: 0.25h — compare DataFrame.columns between training and
#             inference scripts; training order vs alphabetical inference order
# Fix:        0.5h — assert_feature_order() guard using feature_names_in_;
#             reorder inference DataFrame; ColumnTransformer retraining plan
# Validation: 0.25h — AUC-ROC 0.89; edge case assertions pass
# Total:      1.5h

# Retainer engagement cost: 1.5h (estimated $105–165 at mid-level rates)

Related retainer posts

• XGBoost developer on retainer — label-encoded categorical feature — unseen category at inference mapped to -1 — all unseen values routed down same decision tree branch — systematic bias for new segments — 3 weeks
• LightGBM developer on retainer — categorical feature encoded as integer — LightGBM treats as numeric — split on 2.5 meaningless — 3 weeks
• MLflow developer on retainer — model logged without input example — signature inferred from wrong schema — inference type mismatch — 3 weeks