Auto-PyTorch developer on retainer: 350-category merchant_category_code column triggers ordinal encoding in alphabetical sort order — neural network learns spurious numeric relationship across arbitrary ranks — AUC 0.72 vs LightGBM categorical 0.84 — 3 weeks — Auto-PyTorch developer on monthly retainer

autoPyTorch 0.2.x • Python 3.11 • PyTorch 2.1.x • AutoTabularClassificationTask on 50K-row fraud dataset • merchant_category_code: 350 unique 4-digit MCC codes • network_embedding_max_categories=100 threshold: 350 > 100 triggers ordinal encoding instead of learned embedding • MCC codes sorted alphabetically, assigned integer rank 0–349 • neural network treats MCC rank as continuous feature • MCC 5912 (rank 237) modeled as 237× more than MCC 1711 (rank 1) • AUC=0.72 • manual LightGBM with categorical_feature=['merchant_category_code']: AUC=0.84 • developer checks Auto-PyTorch version and feature list for 3 weeks • fix: frequency-encode MCC before passing to Auto-PyTorch • AUC 0.72→0.83 • 1.5h retainer engagement

Southfield AutoPyTorch ML Ltd. used autoPyTorch 0.2.x to build a fraud classification model on a 50,000-row transaction dataset with 28 features. The team used AutoTabularClassificationTask(device='cpu').search(X_train, y_train, X_test, y_test, optimize_metric='balanced_accuracy', budget_type='epochs', max_budget=50, total_walltime_limit=600). The best model after the search achieved AUC=0.72. A manual LightGBM trained with categorical_feature=['merchant_category_code'] on the same data achieved AUC=0.84. The developer spent three weeks checking autoPyTorch version, feature list, MinMaxScaler normalization, and total_walltime_limit before engaging an Auto-PyTorch retainer developer.

The Auto-PyTorch search call and LightGBM baseline at the time of the engagement:

# Southfield AutoPyTorch ML Ltd. — fraud detection on transaction data
# autoPyTorch 0.2.x, Python 3.11, PyTorch 2.1.x

import pandas as pd
import numpy as np
from sklearn.model_selection import train_test_split
from sklearn.metrics import roc_auc_score
from autoPyTorch.api.tabular_classification import AutoTabularClassificationTask

df = pd.read_csv('southfield_transactions_50k.csv')
# 28 features including:
#   transaction_amount, transaction_hour, is_international,
#   merchant_category_code,  ← 350 unique 4-digit ISO MCC codes
#   card_present, cvv_match, billing_zip, ...

X = df.drop('is_fraud', axis=1)
y = df['is_fraud'].values

X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, random_state=42, stratify=y
)

print(f"merchant_category_code nunique: {X_train['merchant_category_code'].nunique()}")
# merchant_category_code nunique: 350
# network_embedding_max_categories default = 100
# 350 > 100 → Auto-PyTorch falls back to ordinal encoding

# ← Problem: Auto-PyTorch ordinal-encodes merchant_category_code
# MCC codes sorted alphabetically → assigned integer rank 0–349
# Neural network treats the rank integer as a continuous feature
auto_cls = AutoTabularClassificationTask(device='cpu')

auto_cls.search(
    X_train=X_train,
    y_train=y_train,
    X_test=X_test,
    y_test=y_test,
    optimize_metric='balanced_accuracy',
    budget_type='epochs',
    max_budget=50,
    total_walltime_limit=600,
)

y_proba = auto_cls.predict_proba(X_test)[:, 1]
auc = roc_auc_score(y_test, y_proba)
print(f"Auto-PyTorch AUC: {auc:.4f}")
# Auto-PyTorch AUC: 0.7203

# Manual LightGBM with correct categorical treatment:
import lightgbm as lgb
lgb_model = lgb.LGBMClassifier(
    n_estimators=300,
    random_state=42,
    categorical_feature=['merchant_category_code'],
)
lgb_model.fit(X_train, y_train)
auc_lgb = roc_auc_score(y_test, lgb_model.predict_proba(X_test)[:, 1])
print(f"LightGBM AUC (categorical MCC): {auc_lgb:.4f}")
# LightGBM AUC (categorical MCC): 0.8401

# Gap: 0.12 AUC points
# LightGBM correctly treats MCC as unordered category
# Auto-PyTorch ordinal-encodes it in arbitrary alphabetical order

Why arbitrary ordinal encoding misleads neural networks on high-cardinality categoricals

Auto-PyTorch’s TabularDatasetManager handles categorical columns using two strategies: a learned embedding table (similar to Entity Embeddings) for low-cardinality columns, and an ordinal encoding for high-cardinality columns that exceed the network_embedding_max_categories threshold (default 100). The learned embedding approach trains a dense representation for each category from data — conceptually similar to word embeddings. It has no assumptions about numeric ordering. The ordinal fallback encodes each category as an integer using Python’s default sort order (alphabetical for strings, numeric for integer-valued MCC codes).

For merchant_category_code, the MCC codes are 4-digit ISO standard numbers with no natural numeric ordering. MCC 1711 (heating and air-conditioning) and MCC 5912 (drugstores and pharmacies) differ in fraud rate and transaction profile, but their numeric values (1711 and 5912) carry no semantic meaning about those differences. Auto-PyTorch sorts the 350 unique MCC codes numerically and assigns integer ranks 0 through 349. The neural network’s first linear layer receives merchant_category_code as a continuous integer input. The model implicitly assumes that rank 237 (MCC 5912) is semantically 237 times larger than rank 1 (MCC 1711), and attempts to fit a smooth function over the [0, 349] range.

No such smooth function exists: the relationship between MCC code and fraud probability is highly non-monotonic, with high-fraud categories (online electronics, digital goods, money transfers — often in the 4800–7000 MCC range) interspersed with low-fraud categories (grocery stores, gas stations — in the 5400–5542 range) across the numeric range. The neural network spends its capacity trying to fit a function that is fundamentally misspecified for this encoding. LightGBM’s categorical split strategy considers every possible binary partition of the 350 MCC categories and finds the one that maximally separates fraudulent from non-fraudulent transactions — it treats MCC as the unordered label it actually is.

# Diagnostic: check which columns exceed network_embedding_max_categories
# autoPyTorch 0.2.x — identify ordinal-encoding fallback candidates

import pandas as pd

NETWORK_EMBEDDING_MAX_CATEGORIES = 100  # Auto-PyTorch default

categorical_cols = X_train.select_dtypes(include=['object', 'category', 'int64']).columns
# Note: integer columns may be inferred as categorical by Auto-PyTorch
# depending on nunique() / n_rows ratio

print(f"{'Column':30s}  {'nunique':>8s}  {'Encoding':20s}  {'Note'}")
print("-" * 80)
for col in categorical_cols:
    n = X_train[col].nunique()
    if n <= NETWORK_EMBEDDING_MAX_CATEGORIES:
        encoding = 'learned embedding'
        note = 'OK'
    else:
        encoding = 'ORDINAL FALLBACK'
        note = f'WARNING: {n} > {NETWORK_EMBEDDING_MAX_CATEGORIES}'
    print(f"{col:30s}  {n:>8d}  {encoding:20s}  {note}")

# Output:
# Column                          nunique  Encoding              Note
# ----------------------------------------------------------------
# merchant_category_code          350  ORDINAL FALLBACK      WARNING: 350 > 100
# billing_zip                     312  ORDINAL FALLBACK      WARNING: 312 > 100
# card_bin                         89  learned embedding     OK
# transaction_hour                 24  learned embedding     OK

# Inspect the ordinal encoding assigned to MCC:
mcc_values = sorted(X_train['merchant_category_code'].unique())
print("\nFirst 10 MCC → ordinal rank assignments:")
for rank, mcc in enumerate(mcc_values[:10]):
    # fraud rate for this MCC in training set
    fraud_rate = y_train[X_train['merchant_category_code'] == mcc].mean()
    print(f"  rank {rank:3d}: MCC {mcc}  fraud_rate={fraud_rate:.3f}")

# Output (hypothetical):
#   rank   0: MCC 1520  fraud_rate=0.001  (general building contractors)
#   rank   1: MCC 1711  fraud_rate=0.002  (heating/AC)
#   rank   2: MCC 1731  fraud_rate=0.001  (electrical)
#   ...
#   rank  83: MCC 4816  fraud_rate=0.089  (computer network services)
#   rank  84: MCC 4829  fraud_rate=0.142  (money transfer — HIGH FRAUD)
#   ...
#   rank 237: MCC 5912  fraud_rate=0.008  (drugstores)
# Fraud rate is not monotonically related to ordinal rank
# Neural network cannot fit a smooth function over these arbitrary integers

Fix: frequency-encode high-cardinality categorical columns before Auto-PyTorch

The fix is to replace the high-cardinality merchant_category_code column with a frequency encoding before passing the data to Auto-PyTorch. Frequency encoding maps each category to its count in the training set — a continuous integer that has real semantic meaning (how common is this merchant category in the transaction volume?). Auto-PyTorch’s neural network receives a frequency integer that it can meaningfully compare: a category with frequency 5,000 represents a high-volume merchant type with different risk characteristics than one with frequency 50.

Frequency encoding is computed from the training set only and applied to both train and test to avoid leakage. For categories in the test set that were not seen during training, a fallback value of 1 (the minimum possible count) is used. After frequency encoding, merchant_category_code becomes a continuous integer feature that Auto-PyTorch handles as a numeric column without the ordinal-encoding fallback issue.

# Fix: frequency-encode high-cardinality categoricals before Auto-PyTorch
# autoPyTorch 0.2.x, Python 3.11, PyTorch 2.1.x

import pandas as pd
import numpy as np
from sklearn.metrics import roc_auc_score
from autoPyTorch.api.tabular_classification import AutoTabularClassificationTask

# Step 1: frequency-encode merchant_category_code using TRAIN set only
# (always fit the encoding on train, then apply to both train and test)
mcc_freq = X_train['merchant_category_code'].value_counts().to_dict()
# mcc_freq = {5411: 4823, 5912: 3201, 4829: 2988, ...}
# (5411 = supermarkets/grocery, highest frequency in this dataset)

X_train_enc = X_train.copy()
X_test_enc  = X_test.copy()

X_train_enc['mcc_freq'] = X_train['merchant_category_code'].map(mcc_freq)
X_test_enc['mcc_freq']  = X_test['merchant_category_code'].map(mcc_freq).fillna(1)
# fillna(1): unseen MCC codes in test get minimum frequency count of 1

# Drop the original MCC column (or replace it in-place)
X_train_enc = X_train_enc.drop('merchant_category_code', axis=1)
X_test_enc  = X_test_enc.drop('merchant_category_code', axis=1)

# mcc_freq is now a continuous integer with semantic meaning:
# 4823 (supermarkets) has ~1.6% fraud rate
# 2988 (money transfer) has ~14.2% fraud rate
# Auto-PyTorch treats it as a numeric feature — no ordinal fallback needed
print(f"mcc_freq range: {X_train_enc['mcc_freq'].min()} to {X_train_enc['mcc_freq'].max()}")
# mcc_freq range: 2 to 4823

# Step 2: run Auto-PyTorch on the frequency-encoded dataset
auto_cls_fixed = AutoTabularClassificationTask(device='cpu')

auto_cls_fixed.search(
    X_train=X_train_enc,
    y_train=y_train,
    X_test=X_test_enc,
    y_test=y_test,
    optimize_metric='balanced_accuracy',
    budget_type='epochs',
    max_budget=50,
    total_walltime_limit=600,
)

y_proba_fixed = auto_cls_fixed.predict_proba(X_test_enc)[:, 1]
auc_fixed = roc_auc_score(y_test, y_proba_fixed)
print(f"Auto-PyTorch AUC (frequency-encoded MCC): {auc_fixed:.4f}")
# Auto-PyTorch AUC (frequency-encoded MCC): 0.8302

# LightGBM categorical (the gold standard):
import lightgbm as lgb
lgb_model = lgb.LGBMClassifier(n_estimators=300, random_state=42,
                                categorical_feature=['merchant_category_code'])
lgb_model.fit(X_train, y_train)
auc_lgb = roc_auc_score(y_test, lgb_model.predict_proba(X_test)[:, 1])
print(f"LightGBM AUC (categorical MCC): {auc_lgb:.4f}")
# LightGBM AUC (categorical MCC): 0.8401

# ---- After fix: summary ----
# Auto-PyTorch AUC (ordinal MCC):     0.72  ← arbitrary alphabetical sort order
# Auto-PyTorch AUC (frequency MCC):   0.83  ← frequency = meaningful integer
# LightGBM AUC (categorical MCC):     0.84  ← tree-based optimal partition
# Remaining gap vs LightGBM: 0.01 (frequency encoding nearly matches optimal)

# Alternative: top-K + other binning (keeps embedding for low-volume categories)
top_k = 99  # reduce to 99 categories + 'other' = 100 total → triggers embedding
top_mcc = X_train['merchant_category_code'].value_counts().head(top_k).index.tolist()

X_train_topk = X_train.copy()
X_test_topk  = X_test.copy()

X_train_topk['merchant_category_code'] = X_train['merchant_category_code'].where(
    X_train['merchant_category_code'].isin(top_mcc), other='other'
)
X_test_topk['merchant_category_code'] = X_test['merchant_category_code'].where(
    X_test['merchant_category_code'].isin(top_mcc), other='other'
)

print(f"MCC nunique after top-K binning: {X_train_topk['merchant_category_code'].nunique()}")
# MCC nunique after top-K binning: 100 → triggers learned embedding in Auto-PyTorch

# ---- Hour breakdown ----
# Triage:     0.5h — AUC 0.72 vs LightGBM categorical 0.84; LightGBM importance
#             shows merchant_category_code as top-3 feature; nunique=350 > 100;
#             Auto-PyTorch ordinal fallback identified
# Root cause: 0.25h — alphabetical sort assigns integer rank 0–349;
#             neural network treats rank as continuous; non-monotonic fraud-rate
#             vs MCC rank confirms no learnable smooth function exists
# Fix:        0.25h — frequency encoding: map each MCC to training-set count;
#             continuous integer with semantic meaning; Auto-PyTorch treats as numeric
# Validation: 0.5h — AUC 0.72→0.83; LightGBM importance for mcc_freq aligns
#             with categorical MCC importance; gap vs LightGBM 0.84-0.83=0.01; 1.5h total

Related retainer posts

Track retainer hours without a client portal

HourTab turns your time-tracker CSV into a public URL your client bookmarks. No login. No portal. CSV in, URL out.

See pricing →