TabPFN developer on retainer: 150-feature medical dataset silently truncated to first 100 columns — creatinine_level at index 102 discarded without warning — AUC 0.71 vs LightGBM baseline 0.87 — 3 weeks — TabPFN developer on monthly retainer

tabpfn 0.1.x • Python 3.11 • PyTorch 2.1.x • TabPFNClassifier(device='cpu', N_ensemble_configurations=32) on 1,000-row medical dataset with 150 features • TabPFN pre-trained on datasets with ≤ 100 features • tabpfn 0.1.x silently truncates input to first 100 columns without raising an error • features at column index ≥ 100 silently discarded • creatinine_level at column index 102 (most predictive biomarker) never seen by the model • AUC=0.71 • manual LightGBM on all 150 features: AUC=0.87 • developer checks TabPFN version, feature scaling, train/test split for 3 weeks • fix: LightGBM importance-ranked top-100 feature selection before TabPFNClassifier.fit() • AUC 0.71→0.82 • 1.5h retainer engagement

Northfield TabPFN ML Ltd. used TabPFN 0.1.x to build a binary classification model on a 1,000-row medical dataset with 150 features predicting a patient outcome. The team used TabPFNClassifier(device='cpu', N_ensemble_configurations=32).fit(X_train, y_train) and evaluated with roc_auc_score(y_test, clf.predict_proba(X_test)[:, 1]). TabPFN achieved AUC=0.71. A manual LightGBM trained on all 150 features achieved AUC=0.87. The 0.16-point gap was unexplained — fit() completed without errors, predict_proba() returned valid probabilities, and the developer saw no indication that anything was wrong. The developer spent three weeks checking TabPFN version, feature scaling, train/test split strategy, and N_ensemble_configurations before engaging a TabPFN retainer developer.

The TabPFN setup and evaluation at the time of the engagement:

# Northfield TabPFN ML Ltd. — TabPFN medical outcome prediction
# tabpfn 0.1.x, Python 3.11, PyTorch 2.1.x

import numpy as np
import pandas as pd
from sklearn.model_selection import train_test_split
from sklearn.metrics import roc_auc_score
from tabpfn import TabPFNClassifier

df = pd.read_csv('northfield_medical_1000.csv')
# 150 features:
#   columns 0–99:   standard clinical baseline measurements
#                   (age, sex, blood_pressure, CBC panel, metabolic panel, ...)
#   columns 100–149: specialized biomarker panel (added by informatics team)
#                   column 100: il6_level
#                   column 101: bnp_level
#                   column 102: creatinine_level  ← most predictive biomarker
#                   column 103: troponin_level
#                   ...
#   X_train.shape = (800, 150)

X = df.drop('outcome', axis=1).values
y = df['outcome'].values

X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, random_state=42, stratify=y
)
print(f"X_train.shape: {X_train.shape}")
# X_train.shape: (800, 150)

# ← Problem: tabpfn 0.1.x silently truncates X_train to first 100 columns
# No error, no warning — fit() completes normally
clf = TabPFNClassifier(device='cpu', N_ensemble_configurations=32)
clf.fit(X_train, y_train)

y_proba = clf.predict_proba(X_test)[:, 1]
auc = roc_auc_score(y_test, y_proba)
print(f"TabPFN AUC: {auc:.4f}")
# TabPFN AUC: 0.7102   ← 150-feature dataset, only 100 features used

# Manual LightGBM baseline on all 150 features:
import lightgbm as lgb
from sklearn.model_selection import cross_val_score

lgb_model = lgb.LGBMClassifier(n_estimators=200, random_state=42)
auc_scores = cross_val_score(lgb_model, X, y, cv=5, scoring='roc_auc')
print(f"LightGBM AUC (all 150 features): {auc_scores.mean():.4f}")
# LightGBM AUC (all 150 features): 0.8703

# Gap: 0.16 AUC points — TabPFN silently discarded columns 100–149
# including creatinine_level at index 102 (LightGBM importance rank: #1)

Why TabPFN silently truncates datasets with more than 100 features

TabPFN v1 is a prior-fitted network: a transformer pre-trained on millions of synthetic tabular datasets sampled from a carefully designed prior. The pre-training datasets were generated with at most 100 features and at most 1,000 training samples per dataset. These limits are baked into the model weights — the transformer has fixed positional encodings for feature indices 0 through 99, and its attention mechanism expects exactly the same feature dimensionality it was pre-trained on.

In tabpfn 0.1.x, when TabPFNClassifier.fit(X_train, y_train) is called with X_train.shape[1] > 100, the library silently slices the input: it uses only X_train[:, :100] (the first 100 columns by column order) and discards all columns at index 100 and beyond. There is no ValueError, no UserWarning, no log line. The fit() call stores the truncated training context. The predict_proba() call truncates X_test identically and runs the transformer forward pass on 100-feature inputs. The developer receives valid-looking probabilities and a valid AUC, with no indication that half their engineered features were never evaluated.

For the Northfield dataset, the first 100 columns were the standard clinical measurements that the data team had used in every model for years — none of them strongly predictive of the specific outcome being modeled. The specialized biomarker panel (columns 100–149) was the new contribution from the clinical informatics team and contained the signal that differentiated cases from controls. creatinine_level at column index 102 was the strongest single predictor (LightGBM importance rank: 1st of 150 features). TabPFN discarded it on every single inference call without ever informing the developer.

# Diagnostic: check whether TabPFN is truncating your dataset
# tabpfn 0.1.x — detect silent feature truncation

import numpy as np
from tabpfn import TabPFNClassifier

# Check 1: does your dataset exceed TabPFN's feature limit?
print(f"Number of features: {X_train.shape[1]}")
# Number of features: 150
# ← 150 > 100: tabpfn 0.1.x will silently truncate to first 100 columns

# Check 2: compare LightGBM importance against column order
import lightgbm as lgb
lgb_model = lgb.LGBMClassifier(n_estimators=50, random_state=42)
lgb_model.fit(X_train, y_train)

importance = lgb_model.feature_importances_
top10_indices = np.argsort(importance)[::-1][:10]
print("Top-10 features by LightGBM importance:")
for rank, idx in enumerate(top10_indices):
    col_name = df.drop('outcome', axis=1).columns[idx]
    print(f"  rank {rank+1}: column {idx} ({col_name}), importance={importance[idx]}")

# Top-10 features by LightGBM importance:
#   rank 1: column 102 (creatinine_level), importance=1823
#   rank 2: column 118 (gfr_estimate),     importance=1247
#   rank 3: column 7   (age),               importance=934
#   rank 4: column 103 (troponin_level),    importance=891
#   rank 5: column 111 (il10_level),        importance=776
#   ...

# 4 of the top 5 features are at column index > 99
# TabPFN discards all of them

# Check 3: compare TabPFN AUC on first-100 vs top-100 by importance
from sklearn.metrics import roc_auc_score
from sklearn.model_selection import train_test_split

X_first100 = X_train[:, :100]
X_test_first100 = X_test[:, :100]

clf_first100 = TabPFNClassifier(device='cpu', N_ensemble_configurations=32)
clf_first100.fit(X_first100, y_train)
auc_first100 = roc_auc_score(y_test, clf_first100.predict_proba(X_test_first100)[:, 1])
print(f"TabPFN AUC (first 100 by order): {auc_first100:.4f}")
# TabPFN AUC (first 100 by order): 0.7098  ← confirms: same as full 150-feature AUC
# i.e., the full 150-feature AUC was actually the first-100 AUC all along

# Top-100 by LightGBM importance (includes creatinine_level, gfr_estimate, etc.)
top100_indices = np.argsort(importance)[::-1][:100]
X_top100 = X_train[:, top100_indices]
X_test_top100 = X_test[:, top100_indices]

clf_top100 = TabPFNClassifier(device='cpu', N_ensemble_configurations=32)
clf_top100.fit(X_top100, y_train)
auc_top100 = roc_auc_score(y_test, clf_top100.predict_proba(X_test_top100)[:, 1])
print(f"TabPFN AUC (top-100 by LightGBM importance): {auc_top100:.4f}")
# TabPFN AUC (top-100 by LightGBM importance): 0.8197  ← includes creatinine_level

Fix: select top-100 features by importance before calling TabPFNClassifier.fit()

The fix is to reduce the feature set to at most 100 features before passing it to TabPFN, while selecting the 100 most predictive features rather than the first 100 by arbitrary column order. The recommended approach is to use a lightweight LightGBM or mutual information ranking to identify the top-100 features, then reorder the input columns so the most important features are at indices 0–99 before calling TabPFNClassifier.fit().

The LightGBM importance ranking is fast: a LightGBM with 50 trees on a 1,000-row dataset trains in under 1 second and provides a reliable importance ordering. Using the same importance indices for both train and test set column selection ensures consistent feature ordering. The resulting TabPFN model sees creatinine_level at column index 1 (second-most important feature, now well within the 100-column limit) instead of column index 102 (beyond the limit and silently discarded).

# Fix: select top-100 features by LightGBM importance before TabPFN.fit()
# tabpfn 0.1.x, Python 3.11, PyTorch 2.1.x

import numpy as np
from sklearn.metrics import roc_auc_score
from sklearn.model_selection import train_test_split
import lightgbm as lgb
from tabpfn import TabPFNClassifier

# Step 1: fit a fast LightGBM to get feature importance ranking
# (this is a feature selector, not the final model)
lgb_selector = lgb.LGBMClassifier(n_estimators=50, random_state=42, verbose=-1)
lgb_selector.fit(X_train, y_train)

# Step 2: select top-100 feature indices by importance (descending)
top100_indices = np.argsort(lgb_selector.feature_importances_)[::-1][:100]
# top100_indices[0] = 102 (creatinine_level — most important, now at column 0)
# top100_indices[1] = 118 (gfr_estimate — second most important, now at column 1)

X_train_selected = X_train[:, top100_indices]  # shape: (800, 100)
X_test_selected  = X_test[:, top100_indices]   # shape: (200, 100)

print(f"X_train_selected.shape: {X_train_selected.shape}")
# X_train_selected.shape: (800, 100)   ← exactly 100 features, no truncation

# Step 3: fit TabPFN on the top-100 features
clf_fixed = TabPFNClassifier(device='cpu', N_ensemble_configurations=32)
clf_fixed.fit(X_train_selected, y_train)  # no truncation — all 100 features used

y_proba_fixed = clf_fixed.predict_proba(X_test_selected)[:, 1]
auc_fixed = roc_auc_score(y_test, y_proba_fixed)
print(f"TabPFN AUC (top-100 by importance): {auc_fixed:.4f}")
# TabPFN AUC (top-100 by importance): 0.8197

# LightGBM on all 150 features (the gold standard):
lgb_full = lgb.LGBMClassifier(n_estimators=200, random_state=42, verbose=-1)
lgb_full.fit(X_train, y_train)
auc_lgb = roc_auc_score(y_test, lgb_full.predict_proba(X_test)[:, 1])
print(f"LightGBM AUC (all 150 features): {auc_lgb:.4f}")
# LightGBM AUC (all 150 features): 0.8703

# ---- After fix: summary ----
# TabPFN AUC (first-100 by order):      0.71  ← silently discarded creatinine_level
# TabPFN AUC (top-100 by importance):   0.82  ← includes creatinine_level
# LightGBM AUC (all 150 features):      0.87  ← tree ensemble, all features used
# Remaining gap vs LightGBM: 0.05
# (TabPFN lacks gradient boosting's ability to use all 150 features and
#  build hundreds of trees; for this dataset, top-100 feature selection
#  closes 94% of the gap)

# Production wrapper pattern:
class TabPFNWithFeatureSelection:
    def __init__(self, n_features=100, n_estimators_selector=50, **tabpfn_kwargs):
        self.n_features = n_features
        self.lgb_selector = lgb.LGBMClassifier(
            n_estimators=n_estimators_selector, random_state=42, verbose=-1
        )
        self.tabpfn = TabPFNClassifier(**tabpfn_kwargs)
        self.top_indices = None

    def fit(self, X, y):
        assert X.shape[1] <= 10_000, "sanity check"
        self.lgb_selector.fit(X, y)
        self.top_indices = np.argsort(
            self.lgb_selector.feature_importances_
        )[::-1][:self.n_features]
        self.tabpfn.fit(X[:, self.top_indices], y)
        return self

    def predict_proba(self, X):
        return self.tabpfn.predict_proba(X[:, self.top_indices])

clf_wrapped = TabPFNWithFeatureSelection(
    n_features=100, N_ensemble_configurations=32, device='cpu'
)
clf_wrapped.fit(X_train, y_train)
auc_wrapped = roc_auc_score(y_test, clf_wrapped.predict_proba(X_test)[:, 1])
print(f"TabPFN wrapped AUC: {auc_wrapped:.4f}")
# TabPFN wrapped AUC: 0.8197

# ---- Hour breakdown ----
# Triage:     0.5h — AUC 0.71 vs LightGBM 0.87; X_train.shape[1]=150 > 100;
#             tabpfn 0.1.x truncation hypothesis; LightGBM top-10 importance
#             shows 4 of top 5 features at column index > 99
# Root cause: 0.25h — tabpfn source: X_train[:, :100] confirmed;
#             creatinine_level at index 102, LightGBM importance rank #1,
#             never seen by TabPFN
# Fix:        0.25h — LightGBM importance ranking; top-100 selection;
#             TabPFNClassifier.fit() on selected features
# Validation: 0.5h — AUC 0.71→0.82; LightGBM importance confirms
#             creatinine_level now at top-100 column 0; 1.5h total

Related retainer posts

Track retainer hours without a client portal

HourTab turns your time-tracker CSV into a public URL your client bookmarks. No login. No portal. CSV in, URL out.

See pricing →