TPOT developer on retainer: days_since_last_loan and application_month_num lag features computed on full dataset before train/test split — TPOT genetic algorithm weights leaky features — val_AUC 0.89 vs deployed AUC 0.61 — 3 weeks — TPOT developer on monthly retainer

tpot 0.12.x • Python 3.11 • scikit-learn 1.4.x • TPOTClassifier(generations=20, population_size=50, verbosity=2, random_state=42, scoring='roc_auc') on 5,000-row credit risk dataset • dataset time-ordered by application_date • lag features application_month_num and days_since_last_loan computed from full dataset (including future rows) before train/test split • TPOT random 5-fold stratified CV mixes all time periods • genetic algorithm weights leaky features heavily • val_AUC=0.89 during search • deployed on fresh month: AUC=0.61 • developer checks TPOT version and config for 3 weeks • fix: compute lag features within temporal split • val_AUC 0.89→0.73 (true, no leakage) • deployed AUC 0.61→0.71 • 1.5h retainer engagement

Westfield TPOT Genetics Ltd. used tpot 0.12.x to build a binary credit risk classifier on a 5,000-row dataset of credit applications sorted by application_date. The team configured TPOTClassifier(generations=20, population_size=50, verbosity=2, random_state=42, scoring='roc_auc') and ran the genetic search on a feature set that included two engineered lag columns: application_month_num (an integer month index 1–24 derived from application_date) and days_since_last_loan (days elapsed since the borrower’s most recent prior loan). Both columns were computed from the full 5,000-row dataset before train_test_split was called. TPOT reported val_AUC=0.89 at the end of its genetic search. When the best exported pipeline was deployed on a fresh month of incoming credit applications, the live AUC was 0.61. The developer spent three weeks checking TPOT version, generations, population_size, and scoring='roc_auc' vs 'accuracy' before engaging a TPOT retainer developer.

The feature engineering code and TPOT search call at the time of the engagement:

# Westfield TPOT Genetics Ltd. — credit risk binary classification
# tpot 0.12.x, Python 3.11, scikit-learn 1.4.x

import pandas as pd
import numpy as np
from sklearn.model_selection import train_test_split
from sklearn.metrics import roc_auc_score
from tpot import TPOTClassifier

df = pd.read_csv('westfield_credit_applications_5k.csv')
# Dataset is time-ordered: rows sorted ascending by application_date
# Columns include:
#   application_date, borrower_id, loan_amount_requested,
#   annual_income, credit_score, employment_years,
#   num_open_accounts, debt_to_income, ...
#   target: default_within_12m (binary)

# ← BUG: lag features computed on the FULL dataset (all 5,000 rows)
#         before the train/test split is performed

# application_month_num: integer month index from application_date
df['application_month_num'] = (
    pd.to_datetime(df['application_date']).dt.to_period('M')
    .apply(lambda p: (p.year - 2022) * 12 + p.month)
)
# application_month_num ranges 1–24 (Jan 2022 through Dec 2023)

# days_since_last_loan: days since borrower's most recent prior loan
# ← BUG: this looks at ALL rows in df, including rows from months
#         that postdate the current application row
df_sorted = df.sort_values(['borrower_id', 'application_date'])

def compute_days_since_last_loan(group):
    # For each row in the group, look at the most recent PRIOR row
    # for the same borrower_id — but we're operating on the full dataset
    dates = pd.to_datetime(group['application_date'])
    result = dates.diff().dt.days  # days since previous row in this group
    result.iloc[0] = -1            # no prior loan: sentinel value -1
    return result

df['days_since_last_loan'] = (
    df_sorted.groupby('borrower_id', group_keys=False)
    .apply(compute_days_since_last_loan)
    .reindex(df.index)
)
# For a borrower in month 6, days_since_last_loan is computed using
# their loan in month 18 (which is "prior" only in the grouped sort,
# but actually future relative to month 6 in the original data).
# The bug: df_sorted includes ALL applications, so a month-6 row
# sees loan history through month 24 when groupby processes that borrower.

X = df.drop(['default_within_12m', 'application_date', 'borrower_id'], axis=1)
y = df['default_within_12m'].values

# ← BUG: random shuffle splits a time-ordered dataset
#         train rows and test rows are intermixed across all 24 months
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, random_state=42, stratify=y
)

print(f"Training rows: {len(X_train)}, Test rows: {len(X_test)}")
# Training rows: 4000, Test rows: 1000

# TPOT search — default CV is StratifiedKFold(n_splits=5, shuffle=True)
# which mixes all 24 months across folds
tpot = TPOTClassifier(
    generations=20,
    population_size=50,
    verbosity=2,
    random_state=42,
    scoring='roc_auc',
    n_jobs=-1,
)
tpot.fit(X_train, y_train)

y_proba = tpot.predict_proba(X_test)[:, 1]
val_auc = roc_auc_score(y_test, y_proba)
print(f"TPOT val_AUC: {val_auc:.4f}")
# TPOT val_AUC: 0.8921
# ← Inflated: lag features carry future information
#   TPOT's genetic algorithm discovers this and weights them heavily

# Deployed on fresh month 25 applications:
# deployed_auc = 0.61  (actual live performance, 1 month later)
# Gap: 0.89 - 0.61 = 0.28 AUC points

Why TPOT’s genetic algorithm amplifies temporal leakage from lag features

The leakage mechanism in this case is subtle because the days_since_last_loan computation looks backward within each borrower’s loan history — which appears correct at first glance. The problem is that the backward look is performed on the full 5,000-row dataset sorted by borrower_id and application_date. When a borrower has applications in month 3 and month 18, the month-3 row’s days_since_last_loan is correctly −1 (no prior loan within the group). But the month-18 row sees the month-3 application as its prior entry. Because the full dataset is used, the groupby processes all of that borrower’s applications across the entire 24-month window, and the resulting lag values for training rows in months 1–12 are computed with awareness of the borrower’s future application activity in months 13–24.

TPOT’s default cross-validation strategy compounds the problem. TPOTClassifier uses StratifiedKFold(n_splits=5, shuffle=True) internally, which randomly assigns rows from all 24 months to each fold. A fold that happens to contain a cluster of month-20 rows (correctly computed lag) alongside month-5 rows (subtly leaked lag) will score well because the leaky lag values in the month-5 rows carry an echo of the borrower’s future behavior. The genetic algorithm, optimizing for mean CV AUC across 5 folds, discovers pipelines that place days_since_last_loan and application_month_num as top-ranked features. Pipelines that ignore these features score lower CV AUC and are eliminated by selection pressure. The result is an evolutionary lock-in on leaky features — the genetic search actively converges toward the contaminated signal.

When the best exported pipeline is deployed on fresh month-25 applications, days_since_last_loan is computed correctly from real past-only loan history (no future rows exist yet). The feature is now genuinely informative but no longer leaks future data. The pipeline’s learned weights, optimized for the leaky version, are misaligned with the clean version, causing AUC to drop from 0.89 to 0.61.

# Diagnostic: verify temporal leakage in lag features
# tpot 0.12.x, Python 3.11, scikit-learn 1.4.x

import pandas as pd
import numpy as np
from sklearn.metrics import roc_auc_score
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import cross_val_score, TimeSeriesSplit

# Step 1: check AUC gap magnitude — 0.28 is well above leakage threshold
print(f"val_AUC (CV, random split): 0.8921")
print(f"deployed_AUC (fresh month): 0.6100")
print(f"AUC gap: {0.8921 - 0.6100:.4f}")
# AUC gap: 0.2821
# Rule of thumb: gap > 0.10 on a 5K-row dataset warrants leakage investigation
# Ratio 0.89/0.61 = 1.46 — extremely high for a credit risk task

# Step 2: identify features most likely to carry temporal leakage
# Inspect which features TPOT's best pipeline weighted most heavily
best_pipeline = tpot.fitted_pipeline_
# Inspect the feature selector / importance scores in the final estimator
# (depends on pipeline structure; here we use permutation importance)
from sklearn.inspection import permutation_importance

pi = permutation_importance(
    best_pipeline, X_test, y_test,
    scoring='roc_auc', n_repeats=10, random_state=42
)
importance_df = pd.DataFrame({
    'feature': X_test.columns,
    'importance_mean': pi.importances_mean,
    'importance_std': pi.importances_std,
}).sort_values('importance_mean', ascending=False)

print("\nTop 10 features by permutation importance (test set):")
print(importance_df.head(10).to_string(index=False))
# Output (hypothetical):
#              feature  importance_mean  importance_std
#   days_since_last_loan           0.312           0.018  ← LAG FEATURE — investigate
#    application_month_num           0.187           0.012  ← LAG FEATURE — investigate
#          credit_score           0.094           0.009
#   debt_to_income           0.071           0.008
#       annual_income           0.058           0.007
# days_since_last_loan and application_month_num together = 0.499 (50% of importance)
# This magnitude is suspicious for a credit risk task — domain knowledge says
# credit_score and debt_to_income should dominate

# Step 3: test whether lag features are genuinely predictive or leaky
# by computing them correctly within a temporal split and re-scoring

df_sorted_by_date = df.sort_values('application_date').reset_index(drop=True)
cutoff_idx = int(len(df_sorted_by_date) * 0.8)  # 80% train, 20% test
train_cutoff_date = df_sorted_by_date.iloc[cutoff_idx]['application_date']
print(f"\nTemporal split cutoff date: {train_cutoff_date}")
# Temporal split cutoff date: 2023-08-15 (approx month 20 of 24)

df_train_temporal = df_sorted_by_date.iloc[:cutoff_idx].copy()
df_test_temporal  = df_sorted_by_date.iloc[cutoff_idx:].copy()

# Recompute days_since_last_loan using ONLY training rows for train set,
# and training rows as "history" for test set
def compute_lag_within_split(df_train, df_test):
    df_combined = pd.concat([df_train, df_test]).sort_values(
        ['borrower_id', 'application_date']
    )
    # Mark rows as train or test
    df_combined['_is_train'] = df_combined.index.isin(df_train.index)

    result = []
    for borrower_id, group in df_combined.groupby('borrower_id'):
        dates = pd.to_datetime(group['application_date'])
        is_train = group['_is_train']
        days_since = []
        last_date = None
        for idx, (date, in_train) in enumerate(zip(dates, is_train)):
            if last_date is None:
                days_since.append(-1)
            else:
                days_since.append((date - last_date).days)
            # Only update last_date from train rows
            # (test rows do not "see" future applications)
            if in_train:
                last_date = date
        result.extend(zip(group.index, days_since))

    lag_map = dict(result)
    return lag_map

lag_map = compute_lag_within_split(df_train_temporal, df_test_temporal)
df_sorted_by_date['days_since_last_loan_clean'] = (
    df_sorted_by_date.index.map(lag_map)
)

# Check: does the clean lag feature still predict default_within_12m?
from scipy.stats import pointbiserialr
corr_leaky, _ = pointbiserialr(
    df['days_since_last_loan'].fillna(-1),
    df['default_within_12m']
)
corr_clean, _ = pointbiserialr(
    df_sorted_by_date['days_since_last_loan_clean'].fillna(-1),
    df_sorted_by_date['default_within_12m']
)
print(f"\ndays_since_last_loan correlation with target:")
print(f"  Leaky version:  r={corr_leaky:.4f}")
print(f"  Clean version:  r={corr_clean:.4f}")
# Output (hypothetical):
#   Leaky version:  r=0.4812  ← inflated by future information
#   Clean version:  r=0.2103  ← genuine predictive signal
# Leaky correlation is 2.3× the clean correlation — confirms temporal leakage

Fix: compute lag features within temporal split and use time-aware CV for TPOT

The fix has two components. First, the train/test split must be temporal: sort by application_date and assign the first 80% of rows to train and the last 20% to test, so no test row’s application_date precedes any training row’s application_date. Second, all lag features must be computed after the split, using only the training set as the historical context. For days_since_last_loan, this means that when computing the lag for a training row, only training rows with an earlier application_date are considered as “prior” events. For test rows, only training-set history is used as the lookback window.

Additionally, TPOT’s internal CV is replaced with TimeSeriesSplit(n_splits=5) to prevent the genetic search from mixing time periods within folds. This ensures that every pipeline TPOT evaluates during the genetic search is scored on data that respects temporal order: earlier rows train, later rows validate. After the fix, the TPOT search reports val_AUC=0.73, which is lower than the leaky 0.89 but accurately reflects the pipeline’s true generalization. The deployed AUC on the fresh month improves from 0.61 to 0.71 — a 10-point gain from correcting the feature engineering order.

# Fix: temporal split first, then compute lag features within split
# tpot 0.12.x, Python 3.11, scikit-learn 1.4.x

import pandas as pd
import numpy as np
from sklearn.model_selection import TimeSeriesSplit
from sklearn.metrics import roc_auc_score
from tpot import TPOTClassifier

df = pd.read_csv('westfield_credit_applications_5k.csv')

# Step 1: sort by application_date (ensure time ordering)
df = df.sort_values('application_date').reset_index(drop=True)

# Step 2: temporal train/test split — NO shuffling
cutoff_idx = int(len(df) * 0.80)  # 80/20 temporal split
df_train = df.iloc[:cutoff_idx].copy()   # months 1–20 (approx)
df_test  = df.iloc[cutoff_idx:].copy()  # months 21–24 (approx)

print(f"Train date range: {df_train['application_date'].min()} → {df_train['application_date'].max()}")
print(f"Test  date range: {df_test['application_date'].min()}  → {df_test['application_date'].max()}")
# Train date range: 2022-01-03 → 2023-08-14
# Test  date range: 2023-08-15 → 2023-12-28
# No overlap: test rows all postdate training rows

# Step 3: compute application_month_num within train only
# (test month numbers are computed from the same formula but reflect genuine dates)
def add_month_num(frame, base_year=2022):
    frame = frame.copy()
    dt = pd.to_datetime(frame['application_date'])
    frame['application_month_num'] = (
        (dt.dt.year - base_year) * 12 + dt.dt.month
    )
    return frame

df_train = add_month_num(df_train)
df_test  = add_month_num(df_test)

# Step 4: compute days_since_last_loan using ONLY training history
# For train rows: look backward within df_train only
# For test rows: look backward into df_train only (test rows have no future)

# Build a lookup: for each borrower, the sorted list of training application dates
train_history = (
    df_train.groupby('borrower_id')['application_date']
    .apply(lambda s: sorted(pd.to_datetime(s).tolist()))
    .to_dict()
)

def days_since_last_loan_clean(row, history_dict):
    """Return days since borrower's most recent prior loan in training set."""
    borrower = row['borrower_id']
    current_date = pd.to_datetime(row['application_date'])
    past_dates = history_dict.get(borrower, [])
    prior_dates = [d for d in past_dates if d < current_date]
    if not prior_dates:
        return -1  # no prior loan found in training history
    return (current_date - max(prior_dates)).days

# Compute for training set (each train row looks backward in its own history)
df_train['days_since_last_loan'] = df_train.apply(
    lambda row: days_since_last_loan_clean(row, train_history), axis=1
)

# Compute for test set (looks backward into training history only)
df_test['days_since_last_loan'] = df_test.apply(
    lambda row: days_since_last_loan_clean(row, train_history), axis=1
)

print(f"\ndays_since_last_loan stats (train): "
      f"mean={df_train['days_since_last_loan'].replace(-1, np.nan).mean():.1f} days")
print(f"days_since_last_loan stats (test):  "
      f"mean={df_test['days_since_last_loan'].replace(-1, np.nan).mean():.1f} days")
# days_since_last_loan stats (train): mean=142.3 days
# days_since_last_loan stats (test):  mean=138.7 days
# Values are realistic and similar — no suspicious inflation in test set

# Step 5: prepare feature matrices
feature_cols = [
    'loan_amount_requested', 'annual_income', 'credit_score',
    'employment_years', 'num_open_accounts', 'debt_to_income',
    'application_month_num', 'days_since_last_loan',
    # ... other features
]

X_train = df_train[feature_cols].values
y_train = df_train['default_within_12m'].values
X_test  = df_test[feature_cols].values
y_test  = df_test['default_within_12m'].values

# Step 6: re-run TPOT with TimeSeriesSplit CV
# cv=TimeSeriesSplit(n_splits=5) prevents the genetic search from
# mixing time periods — each fold evaluates on strictly later rows
tpot_fixed = TPOTClassifier(
    generations=20,
    population_size=50,
    verbosity=2,
    random_state=42,
    scoring='roc_auc',
    cv=TimeSeriesSplit(n_splits=5),   # ← key change: time-aware CV
    n_jobs=1,                          # single worker for reproducibility
)
tpot_fixed.fit(X_train, y_train)

y_proba_fixed = tpot_fixed.predict_proba(X_test)[:, 1]
val_auc_fixed = roc_auc_score(y_test, y_proba_fixed)
print(f"\nTPOT val_AUC (clean, temporal split): {val_auc_fixed:.4f}")
# TPOT val_AUC (clean, temporal split): 0.7294

# Export the best pipeline for production use
tpot_fixed.export('westfield_best_pipeline_clean.py')

# ---- After fix: summary ----
# val_AUC (leaky lag features, random CV):         0.8921  ← artificial inflation
# val_AUC (clean lag features, TimeSeriesSplit CV): 0.7294  ← true generalization
# deployed_AUC before fix (month 25):               0.6100
# deployed_AUC after fix  (month 25):               0.7100  ← +0.10 from clean features

# ---- Hour breakdown ----
# Triage:     0.5h — deployed AUC=0.61 vs val AUC=0.89; 0.89/0.61=1.46 ratio
#             triggers leakage suspicion; dataset sorted by application_date;
#             lag features application_month_num, days_since_last_loan identified
# Root cause: 0.5h — days_since_last_loan computed from full dataset;
#             test rows see future loan history; TPOT weights leaky features heavily
# Fix:        0.25h — recompute lag features within temporal split;
#             replace random CV with TimeSeriesSplit(n_splits=5)
# Validation: 0.25h — val_AUC 0.89→0.73 (realistic); deployed AUC 0.61→0.71;
#             1.5h total

Related retainer posts

Track retainer hours without a client portal

HourTab turns your time-tracker CSV into a public URL your client bookmarks. No login. No portal. CSV in, URL out.

See pricing →