XGBoost developer on retainer: unseen categorical value at inference label-encoded to -1 — XGBoost decision tree routes all -1 records down same branch — systematic wrong predictions for new category — 3 weeks — XGBoost developer on monthly retainer

xgboost 2.0.x • Python 3.11 • binary classification (loan approval) • 50,000-row training set • 15 features • region feature label-encoded {north:0, south:1, east:2, west:3, central:4} • new overseas region appears at inference • LabelEncoder raises KeyError • developer maps unseen to -1 • XGBoost routes all -1 records down left branch at every region split • all overseas applicants land on same leaf • systematic approval rate 0.87 vs 0.32 for comparable domestic segments • false-positive rate 8%→31% • developer checks data quality, feature importance — 3 weeks • fix: map unseen to NaN • XGBoost default direction • 1.5h retainer engagement

Westfield Credit Scoring Ltd. built their loan approval model on XGBoost in August 2026, training on 50,000 historical applications across five domestic regions: north, south, east, west, and central. The region feature was label-encoded to integers 0–4 before training. The model achieved a false-positive rate of 8% on held-out validation data and was deployed to production scoring 500 new applications per day.

Three months after deployment, Westfield expanded to international markets. A new overseas region category began appearing in the inference pipeline. The production LabelEncoder raised a KeyError for the unseen value. To prevent the pipeline from crashing, the developer added a defaultdict fallback that mapped any unknown region to -1. The inference pipeline stopped crashing. But the overall false-positive rate climbed from 8% to 31% over the following two weeks. Overseas applicants were being approved at a rate of 87% compared to 32% for comparable domestic applicants. The developer checked data quality, feature importance scores, and model hyperparameters for three weeks. They engaged an XGBoost retainer developer.

The encoding and inference pipeline at the time of the engagement:

# inference.py
# Westfield Credit Scoring Ltd.
# xgboost 2.0.x, Python 3.11

import xgboost as xgb
import numpy as np
from collections import defaultdict

model = xgb.XGBClassifier()
model.load_model("loan_model.ubj")

# Region encoding from training — 5 known categories
REGION_ENCODING = {
    "north":   0,
    "south":   1,
    "east":    2,
    "west":    3,
    "central": 4,
}

# Developer added this fallback after 'overseas' caused KeyError
region_encoder = defaultdict(lambda: -1, REGION_ENCODING)

def score_application(application: dict) -> float:
    """
    Returns probability of loan approval (0.0 to 1.0).
    """
    region_encoded = region_encoder[application["region"]]
    # For 'overseas': region_encoder["overseas"] → -1
    # -1 is treated by XGBoost as a real integer feature value

    features = np.array([[
        application["income"],
        application["age"],
        application["credit_score"],
        application["years_employed"],
        # ... 10 more features ...
        region_encoded,                  # -1 for overseas applicants
    ]], dtype=np.float32)

    prob = model.predict_proba(features)[0][1]
    return prob

    # XGBoost decision tree splits on region:
    # Tree 0, Node 3: region < 0.5  → left (high-approval leaf, score +1.2)
    #                 region >= 0.5 → right (further splits)
    # Tree 0, Node 7: region < 1.5  → left (moderate-approval leaf, score +0.3)
    # ...
    # For region = -1: -1 < 0.5 TRUE → always routes left at node 3
    # All overseas applicants land on the same high-approval leaf
    # predict_proba returns 0.87 for all overseas applicants regardless of
    # income, age, credit_score, or years_employed

Why XGBoost routes all -1 values down the same tree branch

XGBoost's decision trees use numeric split thresholds learned during training. For a label-encoded feature like region with values 0–4, the split thresholds in the tree are fractional boundaries learned from the training data: typically 0.5, 1.5, 2.5, 3.5. A record routes left if its feature value is less than the threshold, right if it is greater than or equal to the threshold.

When region = -1 is passed at inference, XGBoost evaluates -1 < 0.5, which is True. So the record routes left at the first region split. At the next region split, -1 < 1.5 is also True — left again. Every split threshold in the tree that involves region produces a left-routing decision for -1, because -1 is less than every possible threshold for a feature that had values 0–4 during training. All overseas applicants are routed to the same leaf node across all trees, receiving a deterministic prediction that reflects whatever population happened to dominate the leftmost leaf during training. The prediction is numerically valid (a probability between 0 and 1) and XGBoost raises no error.

This is qualitatively different from the behavior of NaN. XGBoost handles missing values specially: for each split node, during training, it records a default_direction — whether missing values at that node should go left or right — based on which direction produces the better objective on the training data. Missing values at inference follow this learned default direction rather than deterministically routing left. The default direction may be different for different nodes and different trees, producing predictions that reflect the model's best learned behavior for unknown categories rather than an artifact of the encoding fallback value.

# Diagnosing the unseen-category routing — XGBoost tree inspection

import xgboost as xgb
import numpy as np

model = xgb.XGBClassifier()
model.load_model("loan_model.ubj")

# Step 1: inspect a single tree to see region split thresholds
booster = model.get_booster()
tree_df = booster.trees_to_dataframe()

region_splits = tree_df[tree_df["Feature"] == "region"]
print("Region split thresholds:")
print(region_splits[["Tree", "Node", "Split", "Yes", "No", "Missing"]].head(10))
# Tree  Node  Split  Yes   No    Missing
#    0     3    0.5   7     8     7
#    0     7    1.5  14    15     7
#    0    14    2.5  28    29    28
#    1     1    1.5   3     4     3
# ...
# 'Missing' column shows default direction for NaN:
# varies by node — some route to Yes (left), some to No (right)

# Step 2: check predictions for region=-1 vs NaN
income, age, credit_score, years_employed = 75000, 38, 710, 6

row_minus1 = np.array([[income, age, credit_score, years_employed,
                         0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
                         -1]], dtype=np.float32)

row_nan = np.array([[income, age, credit_score, years_employed,
                     0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
                     np.nan]], dtype=np.float32)

prob_minus1 = model.predict_proba(row_minus1)[0][1]
prob_nan    = model.predict_proba(row_nan)[0][1]

print(f"region=-1  → approval prob: {prob_minus1:.3f}")
# region=-1  → approval prob: 0.872   ← deterministic, always this
print(f"region=NaN → approval prob: {prob_nan:.3f}")
# region=NaN → approval prob: 0.334   ← NaN follows default_direction

# Step 3: audit per-region prediction distribution
import pandas as pd
df = pd.read_csv("applications_recent.csv")
df["region_encoded"] = df["region"].map(lambda r: REGION_ENCODING.get(r, -1))
# All overseas rows get -1

predictions = model.predict_proba(df.drop("approved", axis=1).values)[:,1]
df["pred_prob"] = predictions
print(df.groupby("region")["pred_prob"].mean())
# central    0.321
# east       0.318
# north      0.342
# overseas   0.872   ← dramatically higher — encoding artifact
# south      0.309
# west       0.330

The fix: map unseen categories to NaN and let XGBoost's default direction handle them

Replace the -1 fallback with np.nan. XGBoost's default direction mechanism routes NaN values using the per-node direction that was optimal on the training data. This is not guaranteed to be semantically correct for a truly unseen category, but it avoids the systematic bias caused by routing all unseen records through the same deterministic path. For long-term correctness, migrate to XGBoost's native categorical support, which explicitly handles unseen categories at inference.

# inference.py — FIXED
# xgboost 2.0.x, Python 3.11

import xgboost as xgb
import numpy as np

model = xgb.XGBClassifier()
model.load_model("loan_model.ubj")

REGION_ENCODING = {
    "north":   0,
    "south":   1,
    "east":    2,
    "west":    3,
    "central": 4,
}

KNOWN_REGIONS = set(REGION_ENCODING.keys())

def encode_region(region: str) -> float:
    """
    Encode region as integer. Map unseen categories to NaN.
    XGBoost uses the default_direction learned during training for NaN,
    avoiding the systematic bias of a fixed fallback value like -1.
    """
    if region in REGION_ENCODING:
        return float(REGION_ENCODING[region])
    # Log unseen category for monitoring — do not silently swallow
    import logging
    logging.warning(
        f"Unseen region category at inference: {region!r}. "
        f"Known categories: {sorted(KNOWN_REGIONS)}. "
        "Encoding as NaN — XGBoost default direction will be used."
    )
    return np.nan

def score_application(application: dict) -> float:
    region_encoded = encode_region(application["region"])
    # For 'overseas': np.nan
    # XGBoost treats NaN with default_direction — varies by split node
    # Prediction reflects learned behavior, not a deterministic -1 artifact

    features = np.array([[
        application["income"],
        application["age"],
        application["credit_score"],
        application["years_employed"],
        # ... 10 more features ...
        region_encoded,
    ]], dtype=np.float32)

    prob = model.predict_proba(features)[0][1]
    return prob

Long-term fix: native categorical support

For future model versions, use XGBoost's built-in categorical handling. Pass the region column as a pandas Categorical dtype. XGBoost stores the category set at training time and explicitly handles unseen categories at inference without any manual encoding. This is the most robust solution for features where new categories can appear post-deployment.

# training_v2.py — native categorical support
# xgboost 2.0.x, Python 3.11

import xgboost as xgb
import pandas as pd

df = pd.read_csv("applications_train.csv")

# Mark region as a pandas Categorical — XGBoost handles encoding internally
df["region"] = df["region"].astype("category")
# Categories: ['central', 'east', 'north', 'south', 'west']

X = df.drop("approved", axis=1)
y = df["approved"]

model = xgb.XGBClassifier(
    enable_categorical=True,   # use XGBoost native categorical handling
    tree_method="hist",        # required for enable_categorical
    n_estimators=300,
    max_depth=6,
    learning_rate=0.05,
)
model.fit(X, y)
model.save_model("loan_model_v2.ubj")

# Inference with unseen category:
import numpy as np
df_inf = pd.DataFrame([{
    "income": 75000, "age": 38, "credit_score": 710,
    "years_employed": 6,
    # ... 10 more features ...
    "region": "overseas",   # unseen at training time
}])
df_inf["region"] = df_inf["region"].astype("category")

model2 = xgb.XGBClassifier()
model2.load_model("loan_model_v2.ubj")

prob = model2.predict_proba(df_inf)[0][1]
# XGBoost native categorical routing: unseen category follows
# the default_direction at each split — no manual encoding needed

Validation

# Validation log — November 2026
# xgboost 2.0.x, Python 3.11

# ---- Before fix ----
# region encoding: unseen category 'overseas' → -1
# XGBoost routes region=-1 left at every split threshold (0.5, 1.5, 2.5, 3.5)
# All overseas applicants land on same high-approval leaf
# Per-segment approval rate: overseas 0.872 vs domestic mean 0.324
# Overall false-positive rate: 31% (up from 8%)

# ---- After fix: map unseen to NaN ----
# region encoding: unseen category 'overseas' → NaN
# XGBoost applies default_direction per split node
# Per-segment approval rate: overseas 0.334 (comparable to domestic mean 0.324)
# Overall false-positive rate: 8% (returned to baseline)

# ---- Monitoring added ----
# Per-segment prediction audit runs hourly on last 500 predictions
# Alert fires when any segment's mean predicted probability
# deviates > 0.15 from its 7-day rolling mean
# Unseen categories logged with WARNING level — counted by Datadog alert

# ---- Hour breakdown ----
# Triage:     0.5h — false-positive rate 31%; per-segment audit;
#             overseas approval rate 0.87 vs 0.32 for comparable segments
# Root cause: 0.25h — inspect encoding fallback code; defaultdict→-1;
#             XGBoost tree dump shows region split thresholds 0.5–3.5 > -1
# Fix:        0.5h — replace -1 fallback with NaN; add logging for unseen;
#             verify NaN default direction behavior; per-segment alert
# Validation: 0.25h — overseas rate 0.334; FPR 8%; per-region distribution normal
# Total:      1.5h

# Retainer engagement cost: 1.5h (estimated $105–165 at mid-level rates)

Related retainer posts

• scikit-learn developer on retainer — Pipeline StandardScaler transforms columns by position not name — inference DataFrame column order differs from training — predictions silently wrong — 3 weeks
• LightGBM developer on retainer — categorical feature encoded as integer — LightGBM treats as numeric — meaningless splits at non-integer boundaries — 3 weeks
• MLflow developer on retainer — model logged without input example — inferred signature mismatch at inference — predictions fail type validation — 3 weeks