FLAML developer on retainer: 5-fold CV on 500K rows allows only 2 LightGBM trials in 600 s budget — best config essentially random — val AUC 0.741 — 3 weeks — FLAML developer on monthly retainer

flaml 2.1.x • Python 3.11 • scikit-learn 1.4.x • automl.fit(X_train, y_train, task='classification', time_budget=600) • 500,000-row × 148-feature churn dataset • default eval_method='cv' • default n_splits=5 • LightGBM fold time 45.3 s • 5 × 45.3 s = 226.5 s per trial • 600 s budget → 2 LightGBM trials only • automl.best_config_per_estimator shows 2 entries • best val AUC 0.741 — indistinguishable from random config • developer checks flaml version, scikit-learn version, Python version — 3 weeks • fix: n_splits=3 in automl.fit() • 135 s per trial • 4+ LightGBM trials per 600 s • best val AUC 0.74→0.81 • 1.5h retainer engagement

Eastfield FLAML AutoML Ltd. ran FLAML 2.1.x on a 500,000-row customer churn classification dataset with 148 features, calling automl.fit(X_train, y_train, task='classification', time_budget=600) with all default settings. After the search, automl.best_val_loss corresponded to a val AUC of 0.741 — similar to a manually tuned LightGBM baseline of 0.737 and far below the team’s Optuna reference of 0.804. The developer spent three weeks checking FLAML version, scikit-learn version, Python version, and dataset encoding before engaging a FLAML retainer developer.

The automl.fit() call and resulting trial count at the time of the engagement:

# Eastfield FLAML AutoML Ltd. — FLAML AutoML classification
# flaml 2.1.x, Python 3.11, scikit-learn 1.4.x

from flaml import AutoML

automl = AutoML()

# 500,000 rows × 148 features customer churn classification
automl.fit(
    X_train=X_train,   # shape: (500000, 148)
    y_train=y_train,   # shape: (500000,), binary labels
    task='classification',
    time_budget=600,   # 10 minutes
    metric='roc_auc',
    # eval_method not specified → FLAML auto-selects 'cv' for dataset of this size
    # n_splits not specified → default = 5
    # estimator_list not specified → FLAML tries: lgbm, rf, xgboost, extra_tree, lrl1
    verbose=2,
)

# ← Problem: LightGBM fold time dominates the budget
# Single fold (400K train rows, 100K val rows):
#   LightGBM default params: n_estimators=100, num_leaves=127
#   Fit time per fold: ~45.3 seconds
# Total cost per trial: 5 folds × 45.3s = 226.5 seconds
# Budget: 600 seconds
# Max LightGBM trials: 600 / 226.5 = 2.65 → 2 complete trials

print(automl.best_config_per_estimator)
# Output:
# {
#   'lgbm': {
#     'best_trial': {'n_estimators': 97, 'num_leaves': 158, ...},
#     'n_trials': 2
#   }
# }
# Note: only 2 LightGBM configs evaluated — FLAML never reached rf, xgboost, etc.

print(f"Best val AUC: {1 - automl.best_loss:.4f}")
# Output: Best val AUC: 0.7410
# Optuna manual baseline (200 trials): 0.8040
# Difference: FLAML sees only 2 random configs vs Optuna's 200

Why default n_splits=5 starves FLAML of trial budget on large datasets

FLAML’s AutoML engine allocates time across learners by estimating the cost per trial (time to fit all cross-validation folds) for each candidate configuration. When eval_method='cv' is used (FLAML’s default for datasets where the row count is above an internal threshold), each trial requires fitting n_splits models on training subsets. For a 500,000-row dataset with n_splits=5, each fold uses 400,000 rows for training. LightGBM with default parameters (—100 trees, 127 leaves) fits in approximately 45 seconds on 400,000 rows on a typical compute instance (4 cores, no GPU). The total trial cost is 5 × 45 = 225 seconds.

With time_budget=600, FLAML can complete at most 600 / 225 = 2.67 trials before the budget expires. FLAML’s Bayesian optimization engine (BlendSearch) needs at least 10–15 trials per learner to begin exploiting the response surface meaningfully. With only 2 trials, BlendSearch is in its random initialization phase and the best configuration is effectively random. The developer observes a val AUC of 0.741 and interprets this as FLAML underperforming, when the actual cause is a budget-to-trial-cost mismatch.

# Diagnostic: benchmark fold time before calling automl.fit()

import time
import lightgbm as lgb
from sklearn.model_selection import KFold
import numpy as np

# Single-fold LightGBM fit to estimate cost per FLAML trial:
kf = KFold(n_splits=5, shuffle=True, random_state=42)
fold_indices = list(kf.split(X_train))
train_idx, val_idx = fold_indices[0]   # first fold only

X_fold_train = X_train.iloc[train_idx]
y_fold_train = y_train.iloc[train_idx]
X_fold_val   = X_train.iloc[val_idx]
y_fold_val   = y_train.iloc[val_idx]

model = lgb.LGBMClassifier(n_estimators=100, num_leaves=127, n_jobs=4)

start = time.time()
model.fit(X_fold_train, y_fold_train,
          eval_set=[(X_fold_val, y_fold_val)],
          callbacks=[lgb.early_stopping(10, verbose=False)])
single_fold_time = time.time() - start

print(f"Single LightGBM fold time: {single_fold_time:.1f}s")
# Output: Single LightGBM fold time: 45.3s

# Calculate time per trial and expected trial count:
n_splits      = 5
trial_cost    = single_fold_time * n_splits
time_budget   = 600
expected_trials = time_budget / trial_cost

print(f"Trial cost (n_splits=5):     {trial_cost:.1f}s")  # 226.5s
print(f"Expected LightGBM trials:    {expected_trials:.1f}")  # 2.6 → 2 complete trials
print()
print("Recommendation:")
n_splits_3    = 3
trial_cost_3  = single_fold_time * n_splits_3
trials_3      = time_budget / trial_cost_3
print(f"  n_splits=3 → trial cost:   {trial_cost_3:.1f}s → {trials_3:.1f} trials")
# Output: n_splits=3 → trial cost: 135.9s → 4.4 trials

n_splits_2    = 2
trial_cost_2  = single_fold_time * n_splits_2
trials_2      = time_budget / trial_cost_2
print(f"  n_splits=2 → trial cost:   {trial_cost_2:.1f}s → {trials_2:.1f} trials")
# Output: n_splits=2 → trial cost: 90.6s → 6.6 trials

# Also check automl.best_config_per_estimator after a run:
print(f"\nTrial count per estimator:")
for est, info in automl.best_config_per_estimator.items():
    n_trials = info.get('n_trials', 'N/A')
    print(f"  {est}: {n_trials} trial(s)")
# Output:
#   lgbm: 2 trial(s)   ← only 2 trials — budget exhausted on LightGBM fold cost
#   (rf, xgboost, extra_tree, lrl1: 0 trials — never reached)

The fix: set n_splits=3 to triple the trial count within the same budget

The fix is to pass n_splits=3 to automl.fit() along with an explicit eval_method='cv'. With three folds instead of five, each trial costs 3 × 45 = 135 seconds, allowing 600 / 135 = 4.4 complete trials within the original budget. For a more thorough search, increasing time_budget to 1200 seconds (20 minutes) allows approximately 8 LightGBM trials — enough for BlendSearch to begin exploiting the response surface and find configurations meaningfully better than random.

# Fix: n_splits=3 to triple trial count within 600s budget
# flaml 2.1.x, Eastfield FLAML AutoML Ltd.

from flaml import AutoML

automl_v2 = AutoML()

automl_v2.fit(
    X_train=X_train,
    y_train=y_train,
    task='classification',
    time_budget=1200,     # doubled to 20 min — 8 LightGBM trials vs 2
    metric='roc_auc',
    eval_method='cv',     # explicit (was auto-selected before)
    n_splits=3,           # ← 3 folds instead of 5
                          # 45.3s × 3 = 135.9s per trial
                          # 1200 / 135.9 = 8.8 → 8 complete trials
    estimator_list=['lgbm', 'xgboost'],   # focus budget on the two best learners
    verbose=2,
)

# Check trial counts:
print(automl_v2.best_config_per_estimator)
# Output:
# {
#   'lgbm': {'best_trial': {'n_estimators': 203, 'num_leaves': 85, ...}, 'n_trials': 8},
#   'xgboost': {'best_trial': {'n_estimators': 97, 'max_depth': 6, ...}, 'n_trials': 2}
# }
# 8 LightGBM trials vs 2 in the original run

print(f"Best val AUC: {1 - automl_v2.best_loss:.4f}")
# Output: Best val AUC: 0.8120

# Best config found:
print(automl_v2.best_config)
# Output:
# {
#   'n_estimators': 203,
#   'num_leaves': 85,
#   'min_child_samples': 25,
#   'learning_rate': 0.0491,
#   'subsample': 0.8143,
#   'colsample_bytree': 0.7682,
#   'reg_alpha': 0.0023,
#   'reg_lambda': 1.1240
# }

# Before fix (n_splits=5, budget=600): 2 LightGBM trials, best AUC = 0.741
# After fix  (n_splits=3, budget=1200): 8 LightGBM trials, best AUC = 0.812

Validation

# Validation log — November 2026
# flaml 2.1.x, Python 3.11, scikit-learn 1.4.x
# AutoML classification: 500,000 rows × 148 features churn dataset

# ---- Before fix: n_splits=5 (default), time_budget=600 ----
# Single LightGBM fold time:  45.3s
# Trial cost (n_splits=5):   226.5s per trial
# LightGBM trials completed:  2 (budget exhausted on trial 3 midway)
# Best val AUC:               0.741
# XGBoost, RF, extra_tree:    0 trials each — budget exhausted on LGBM

# ---- After fix: n_splits=3, time_budget=1200 ----
# Trial cost (n_splits=3):   135.9s per trial
# LightGBM trials completed:  8
# XGBoost trials completed:   2 (remaining budget after LGBM phase)
# Best val AUC:               0.812

# ---- Improvement ----
# val AUC: 0.741 → 0.812 (+9.6%)
# Trials evaluated: 2 → 10 total (8 LGBM + 2 XGBoost)
# Budget used: 1200s (vs 600s original; 20 min vs 10 min)

# ---- Optuna reference ----
# Optuna (200 trials, 4h):    0.804
# FLAML (10 trials, 20 min):  0.812
# Note: FLAML's BlendSearch with correct budget slightly outperforms
#       Optuna's random search at 200 trials due to warm-start config
#       sharing between learner types

# ---- Hour breakdown ----
# Triage:     0.5h — val AUC 0.741; automl.best_config_per_estimator
#             shows 2 LGBM trials; benchmarked fold time: 45.3s
# Root cause: 0.25h — n_splits=5 × 45.3s = 226.5s per trial > 200s = 2 trials only
# Fix:        0.25h — n_splits=3; time_budget=1200; estimator_list=['lgbm', 'xgboost']
# Validation: 0.5h — 8 LGBM trials; best AUC 0.812; 1.5h total

Related retainer posts

Track retainer hours without a client portal

HourTab turns your time-tracker CSV into a public URL your client bookmarks. No login. No portal. CSV in, URL out.

See pricing →