LightGBM developer on retainer: num_leaves=2048 min_data_in_leaf=1 produces 0.999 train AUC and 0.61 validation AUC — 6 months of learning_rate tuning doesn’t fix it — fix: num_leaves=63 min_data_in_leaf=100 — validation AUC 0.61→0.89 — LightGBM developer on monthly retainer

LightGBM 4.1.0 • Python 3.11 • AWS EC2 r6i.4xlarge • 240,000-row credit risk dataset • 47 features • num_leaves=2048 • min_data_in_leaf=1 • structural overfitting • train AUC 0.999 • validation AUC 0.61 • 6 months of learning_rate tuning • fix: num_leaves=63 min_data_in_leaf=100 • validation AUC 0.89 • retainer billing

On October 14, 2026, a new ML engineer joined Riverton Credit Analytics Inc. on a consulting engagement and asked a question that had been unasked for six months: why did the credit risk scoring model have a training AUC of 0.999 and a validation AUC of 0.61? The team had been running LightGBM tuning experiments since April 2026, systematically adjusting learning_rate (from 0.1 down to 0.005), n_estimators (from 100 up to 2,000 with early stopping), subsample, and colsample_bytree. Across 18 separate experiments, validation AUC had never exceeded 0.63. The new engineer opened the LightGBM documentation, read the description of num_leaves, and noticed that the team’s configuration used num_leaves=2048 — 66 times the default value of 31. The LightGBM retainer developer was engaged on October 15 to confirm the diagnosis and implement the fix. Changing num_leaves to 63 and min_data_in_leaf to 100 brought validation AUC from 0.61 to 0.89 in a single retraining run.

Riverton Credit Analytics ran a credit risk scoring pipeline for a regional lending institution. The model assessed default probability for consumer loan applications using a LightGBM gradient boosting classifier trained on 240,000 historical loan records with 47 engineered features (credit utilization ratio, payment history length, derogatory mark count, inquiry count, account age distribution, debt-to-income ratio, and derived features). The dataset had a 12.8% positive rate (default events). The model was trained on AWS EC2 r6i.4xlarge (16 vCPU, 128 GB RAM) using LightGBM 4.1.0 and Python 3.11. The production deployment required a model with AUC above 0.85 to meet the lending institution’s risk management standards. As of October 2026, the deployed model was scoring at 0.61 AUC on held-out test data — a coin flip for default prediction.

The original LightGBM configuration had been set in April 2026 by a data scientist who had copied parameters from a Kaggle competition notebook. The notebook had been written for a dataset with 10 million rows, where num_leaves=2048 and min_data_in_leaf=1 were part of an aggressive ensemble configuration tuned for that specific data scale. Applied to Riverton’s 240,000-row dataset, the parameters produced a model that memorized the training set rather than learning generalizable credit risk patterns.

# credit_model.py — Riverton Credit Analytics Inc.
# LightGBM 4.1.0, Python 3.11, scikit-learn API
# Configuration set April 2026; copied from Kaggle notebook for 10M-row dataset.

import lightgbm as lgb
from sklearn.model_selection import train_test_split
from sklearn.metrics import roc_auc_score

X_train, X_val, y_train, y_val = train_test_split(
    X, y, test_size=0.2, random_state=42, stratify=y
)
# Training set: 192,000 rows, 12.8% positive rate
# Validation set: 48,000 rows, 12.8% positive rate

model = lgb.LGBMClassifier(
    # BUG: num_leaves=2048 is 66× the LightGBM default of 31.
    # With 240,000 training rows, num_leaves=2048 creates trees
    # with up to 2,048 leaf nodes. LightGBM builds leaf-wise
    # (best-first) trees: each leaf is split by choosing the
    # split with the highest gain among all current leaves.
    # With num_leaves=2048 and min_data_in_leaf=1, the tree
    # continues splitting until it has 2,048 leaves, each of
    # which can contain as few as ONE training example.
    # A single training example per leaf = the tree memorizes
    # the training label for that example exactly.
    num_leaves=2048,          # ← THE BUG (should be 31–127 for this dataset)
    min_data_in_leaf=1,       # ← THE BUG (should be 50–200 for this dataset)
    learning_rate=0.05,
    n_estimators=500,
    subsample=0.8,
    colsample_bytree=0.8,
    random_state=42
)

model.fit(
    X_train, y_train,
    eval_set=[(X_val, y_val)],
    callbacks=[lgb.early_stopping(50)]
)

train_auc = roc_auc_score(y_train, model.predict_proba(X_train)[:, 1])
val_auc = roc_auc_score(y_val, model.predict_proba(X_val)[:, 1])
print(f"Train AUC: {train_auc:.4f}")   # 0.9994
print(f"Val   AUC: {val_auc:.4f}")     # 0.6107

# 6 months of tuning experiments (April–October 2026), none of which helped:
# Experiment 1:  learning_rate=0.01  → val AUC 0.6124
# Experiment 2:  learning_rate=0.001 → val AUC 0.6089 (worse)
# Experiment 3:  n_estimators=2000   → val AUC 0.6118
# Experiment 4:  subsample=0.6       → val AUC 0.6131
# Experiment 5:  colsample_bytree=0.6 → val AUC 0.6098
# ...18 experiments total. Best validation AUC ever achieved: 0.6289.
# Learning_rate and n_estimators control convergence speed and depth,
# but they cannot fix structural overfitting from num_leaves=2048.
# With 2048-leaf trees, the model already memorizes training labels
# before the learning_rate has a chance to slow convergence.

LightGBM uses leaf-wise (best-first) tree growth, which differs from the level-wise growth used by XGBoost. In leaf-wise growth, the algorithm selects the leaf with the highest gain and splits it — regardless of which level of the tree that leaf is at. This produces asymmetric trees that can be more accurate than level-wise trees for the same number of leaves. The num_leaves parameter directly controls the maximum number of leaves per tree. A tree with 2,048 leaves and min_data_in_leaf=1 can partition the training data into 2,048 groups, each containing as few as one training example. With 192,000 training rows and up to 2,048 leaves per tree, the average leaf contains only 93.75 rows — but min_data_in_leaf=1 allows much smaller leaves, including leaves with a single training example that encodes exactly one label.

The early stopping callback (early_stopping(50)) terminated training at the optimal round for the current configuration. But the “optimal” round was already inside a fully overfit model: by round 50, the trees had memorized the training data well enough that subsequent rounds added marginal training AUC improvement while validation AUC degraded. Early stopping prevented even further overfitting but could not undo the structural memorization already embedded in the first 50–100 rounds. Reducing learning_rate slowed the rate at which the ensemble converged to its memorized training representation — but given enough rounds (which early stopping provided), the model converged to the same overfit state regardless of learning_rate.

# Diagnostic: examine leaf count distribution and feature importance
# Run October 15, 2026 as part of LightGBM retainer investigation

import lightgbm as lgb
import numpy as np

# Load the trained overfit model
model = lgb.Booster(model_file="credit_model_overfit.txt")

# Check num_leaves configuration
print(f"num_leaves: {model.params['num_leaves']}")        # 2048
print(f"min_data_in_leaf: {model.params['min_data_in_leaf']}")  # 1
print(f"num_trees: {model.num_trees()}")                   # 500

# Check actual leaf count distribution across trees
num_leaves_per_tree = []
for tree_idx in range(model.num_trees()):
    tree_info = model.dump_model()['tree_info'][tree_idx]
    leaf_count = len([node for node in _flatten_nodes(tree_info)
                     if node.get('leaf_value') is not None])
    num_leaves_per_tree.append(leaf_count)

print(f"Mean leaves per tree: {np.mean(num_leaves_per_tree):.1f}")   # 2045.7
print(f"Pct trees with 2048 leaves: {np.mean([l == 2048 for l in num_leaves_per_tree]):.1%}")
# 89.1% of trees use exactly 2,048 leaves — confirming the model
# consistently hits the leaf budget on nearly every tree.

# Compare train vs. validation AUC learning curves
results = {}
model.fit(
    X_train, y_train,
    eval_set=[(X_train, y_train), (X_val, y_val)],
    eval_names=['train', 'val'],
    callbacks=[lgb.record_evaluation(results), lgb.early_stopping(50)]
)
# Key finding: train AUC reaches 0.999 by round 23.
# Validation AUC peaks at 0.613 at round 47, then slowly degrades.
# The 0.388 AUC gap between train and validation at round 47
# is characteristic of memorization, not underfitting.
# A correctly configured model (num_leaves=63) shows a gap of <0.05.

Why learning_rate and n_estimators cannot fix structural overfitting from num_leaves

Overfitting in gradient boosting trees has two distinct structural causes: too many boosting rounds (tunable with n_estimators and early stopping) and too many leaves per tree (structural, determined by num_leaves and min_data_in_leaf). The two causes have different remedies. If a model overfits because it runs too many boosting rounds, reducing n_estimators or adding early stopping helps — the model stops accumulating residual corrections before it memorizes noise. If a model overfits because each tree has too many leaves, no adjustment to n_estimators or learning_rate helps, because the structural capacity to memorize is baked into the tree topology: a 2,048-leaf tree with min_data_in_leaf=1 can memorize individual training labels in its first boosting round.

Reducing learning_rate from 0.1 to 0.005 makes the model learn more slowly — each boosting round contributes a smaller correction to the ensemble. But the model is still adding 2,048-leaf trees with min_data_in_leaf=1; it just takes more rounds to memorize the training data. With early stopping enforcing the optimal round, the model terminates at the same structural endpoint regardless of learning_rate: the point where validation AUC stops improving, which for a 2,048-leaf model is around validation AUC 0.61. Regularization parameters (reg_alpha L1, reg_lambda L2) partially constrain leaf values but do not prevent the tree from growing 2,048 leaves. Only num_leaves and min_data_in_leaf control the structural capacity of individual trees.

The fix: num_leaves=63 and min_data_in_leaf=100

The fix was a single hyperparameter change. num_leaves was reduced from 2,048 to 63 (twice the default of 31, appropriate for a 240,000-row dataset with 47 features). min_data_in_leaf was increased from 1 to 100 (requiring at least 100 training examples in each leaf — approximately 0.05% of the training set per leaf, a reasonable constraint for a credit risk model). With num_leaves=63 and min_data_in_leaf=100, the maximum number of distinct leaf partitions that could be created across the entire 500-tree ensemble was 63 × 500 = 31,500 — much smaller than the 2,048 × 500 = 1,024,000 possible partitions in the overfit configuration. The model was no longer structurally capable of memorizing individual training examples.

# credit_model.py — fixed version, deployed October 15, 2026

import lightgbm as lgb
from sklearn.model_selection import train_test_split
from sklearn.metrics import roc_auc_score

X_train, X_val, y_train, y_val = train_test_split(
    X, y, test_size=0.2, random_state=42, stratify=y
)

model = lgb.LGBMClassifier(
    # FIX 1: num_leaves=63 — LightGBM documentation recommends
    # num_leaves < 2^max_depth. For max_depth=-1 (unlimited), a
    # practical guideline is num_leaves ≈ 2–4× sqrt(n_samples).
    # For 192,000 training rows: 2 × sqrt(192,000) ≈ 877.
    # However, num_leaves=63 is a conservative starting point
    # that the LightGBM documentation explicitly recommends for
    # datasets up to ~1M rows. For credit risk with 240K rows, 63
    # balances complexity with generalization.
    num_leaves=63,            # FIX (was 2048)
    # FIX 2: min_data_in_leaf=100 — requires at least 100 training
    # samples per leaf. With 192,000 training rows and num_leaves=63,
    # the average leaf contains ~3,048 samples. min_data_in_leaf=100
    # prevents very small leaves from forming on noisy data splits.
    min_data_in_leaf=100,     # FIX (was 1)
    learning_rate=0.05,
    n_estimators=1000,        # Increased to allow more rounds with lower capacity
    subsample=0.8,
    colsample_bytree=0.8,
    random_state=42
)

model.fit(
    X_train, y_train,
    eval_set=[(X_val, y_val)],
    callbacks=[lgb.early_stopping(50)]
)

train_auc = roc_auc_score(y_train, model.predict_proba(X_train)[:, 1])
val_auc = roc_auc_score(y_val, model.predict_proba(X_val)[:, 1])
print(f"Train AUC: {train_auc:.4f}")   # 0.8472
print(f"Val   AUC: {val_auc:.4f}")     # 0.8914
print(f"Early stopping at round: {model.best_iteration_}")  # 782

# Production scoring on held-out test set:
# Previous (overfit) model: test AUC 0.612
# Fixed model:              test AUC 0.883
# Mean leaves per tree with fixed config:
#   np.mean([...leaf counts...]) → 62.4 (was 2045.7)
# Train/validation AUC gap: 0.847 - 0.891 = -0.044 (train < val at optimal
#   stopping point — model generalizes well with appropriate capacity)

The fixed model trained to completion in approximately 45 minutes (782 rounds with early stopping, compared to 500 rounds for the overfit configuration). The training AUC was 0.847 — far from the near-perfect 0.999 of the overfit model, but the validation AUC was 0.891 and the held-out test set AUC was 0.883. This exceeded the lending institution’s 0.85 AUC threshold for production deployment. The credit risk model was promoted to production within the same retainer session, replacing the six-month-old model that had been scoring at 0.61 AUC.

Why num_leaves=2048 is a recurring LightGBM retainer pattern

The num_leaves=2048 footgun originates from two sources. First, Kaggle competition notebooks for large datasets (10M+ rows) often feature num_leaves in the hundreds or thousands because those datasets have enough data to support many leaves without memorization. Practitioners copying these configurations to smaller enterprise datasets inherit the overfitting risk. Second, LightGBM’s leaf-wise tree growth makes the effect of num_leaves less intuitive than the max_depth parameter in XGBoost or scikit-learn’s gradient boosting. A developer who understands that “deeper trees overfit more” may not realize that more leaves (without a depth constraint) has the same effect in LightGBM’s leaf-wise growth paradigm.

The six-month delay in diagnosing the root cause reflects a common investigation path: when validation AUC is low, developers first adjust convergence parameters (learning_rate, n_estimators) and regularization (subsample, colsample_bytree, reg_alpha, reg_lambda). These adjustments have visible effects on training curves, which creates an illusion of progress. Only a developer who explicitly compares num_leaves against the LightGBM documentation’s recommendation and the dataset size catches the structural overfitting before exhausting the convergence-parameter search space. A LightGBM retainer developer who runs an “AUC gap diagnostic” at onboarding — computing train AUC minus validation AUC and flagging any gap greater than 0.1 as potential structural overfitting — catches this class of bug before six months of fruitless tuning experiments.

The 5.0-hour engagement covered investigation (2.0h — 0.5h review of 18 tuning experiment logs: learning_rate and n_estimators adjustments, no improvement in validation AUC above 0.63; 0.5h LightGBM documentation review: num_leaves=2048 is 66× default, min_data_in_leaf=1 allows single-example leaves; 0.5h leaf count distribution analysis: 89.1% of trees use exactly 2,048 leaves; 0.5h training curve inspection: train AUC reaches 0.999 by round 23, validation AUC peaks at 0.613 at round 47 — characteristic of structural memorization), fix implementation (2.0h — 0.5h new configuration: num_leaves=63, min_data_in_leaf=100, n_estimators=1000, learning_rate=0.05; 1.0h full retraining run: 45 minutes, early stopping at round 782; 0.5h train AUC 0.847, validation AUC 0.891 verified; mean leaves per tree: 62.4 vs. previous 2,045.7), and validation (1.0h — 0.5h held-out test set scoring: test AUC 0.883 vs. previous 0.612; 0.25h SHAP feature importance comparison: fixed model credit_utilization_ratio SHAP values concentrated (low std dev) vs. overfit model (high std dev indicating noise memorization); 0.25h production deployment and score distribution monitoring).

In a HourTab retainer log, the incident session reads: “LightGBM credit risk model overfitting investigation: LightGBM 4.1.0, Python 3.11, AWS EC2 r6i.4xlarge; 240,000-row dataset, 47 features, 12.8% positive rate; num_leaves=2048 (66× default 31), min_data_in_leaf=1; 89.1% of 500 trees use exactly 2,048 leaves; model memorizes training labels: train AUC 0.9994, validation AUC 0.6107; 18 tuning experiments (April–October 2026) adjusting learning_rate, n_estimators, subsample, colsample_bytree — best validation AUC 0.6289; fix: num_leaves=63, min_data_in_leaf=100; retrained: train AUC 0.847, validation AUC 0.891, test AUC 0.883; meets production threshold of 0.85; deployed; 5.0h.”

HourTab turns a time-tracker CSV into a public retainer-hours URL your client can bookmark — no login, no portal, no “how many hours do I have left?” emails. Free for one retainer. $9/mo for up to ten.

FAQ: LightGBM developer retainers

See also: Kubeflow developer retainer · MLflow developer retainer · W&B developer retainer