HPBandSter developer on retainer: BOHB min_budget=10 violates multi-fidelity assumption on correlated feature pair income / income_verified — Spearman rank correlation 0.31 — val_AUC 0.74 vs grid search 0.81 — 3 weeks — HPBandSter developer on monthly retainer

hpbandster 0.7.x • Python 3.11 • PyTorch 2.1.x (used for XGBoost HPO) • BOHB optimizer on 20K-row binary classification • 50 total evaluations • min_budget=10 (10% of data = 2,000 rows training) • max_budget=100 (100% of data = 20,000 rows) • income / income_verified Pearson r=0.94 correlated pair • feature importance flips between 2K and 20K rows • max_features=0.5 config promoted incorrectly by BOHB • val_AUC=0.74 • fixed grid search at max_budget=100: AUC=0.81 • developer checks hpbandster version, eta, workers for 3 weeks • fix: min_budget 10→50 • Spearman rank correlation 0.31→0.87 • val_AUC 0.74→0.80 • 1.5h retainer engagement

Eastfield HPBandSter BO Ltd. used hpbandster 0.7.x to run BOHB hyperparameter optimization for an XGBoost binary classifier on a 20,000-row lending dataset. The team configured BOHB with 50 total evaluations, min_budget=10 (training on 10% of data, 2,000 rows), and max_budget=100 (training on 100% of data, 20,000 rows). The best configuration found by BOHB achieved val_AUC=0.74. A fixed grid search over the same hyperparameter space, evaluated only at max_budget=100, achieved val_AUC=0.81. BOHB is supposed to outperform grid search by intelligently allocating compute — a result 7 AUC points below grid search meant the multi-fidelity assumption had broken down entirely. The developer spent three weeks checking the hpbandster version, the min_budget value, the eta parameter, and the number of BOHB workers before engaging an HPBandSter retainer developer.

The BOHB configuration and XGBoost HPO setup at the time of the engagement:

# Eastfield HPBandSter BO Ltd. — XGBoost HPO on lending binary classification
# hpbandster 0.7.x, Python 3.11

import numpy as np
import pandas as pd
import xgboost as xgb
from sklearn.model_selection import train_test_split
from sklearn.metrics import roc_auc_score
import ConfigSpace as CS
import ConfigSpace.hyperparameters as CSH
import hpbandster.core.nameserver as hpns
import hpbandster.core.result as hpres
from hpbandster.optimizers import BOHB
from hpbandster.core.worker import Worker

df = pd.read_csv('eastfield_lending_20k.csv')
# 20,000 rows, 18 features including:
#   annual_income, loan_amount, dti_ratio,
#   income,           ← raw reported income (numeric, $)
#   income_verified,  ← loan officer verification judgment (numeric, 0–1 score)
#   credit_score, employment_length, ...
# income and income_verified: Pearson r = 0.94

X = df.drop('default', axis=1)
y = df['default'].values

X_train, X_val, y_train, y_val = train_test_split(
    X, y, test_size=0.2, random_state=42, stratify=y
)

# BOHB Worker: trains XGBoost on a fraction of training data
# budget is an integer 10–100 representing % of training rows to use
class XGBWorker(Worker):
    def __init__(self, X_train, y_train, X_val, y_val, **kwargs):
        super().__init__(**kwargs)
        self.X_train = X_train
        self.y_train = y_train
        self.X_val   = X_val
        self.y_val   = y_val

    def compute(self, config, budget, **kwargs):
        # budget: 10 → use 10% of training rows (2,000 rows)
        # budget: 100 → use 100% of training rows (20,000 rows)
        n_rows = int(len(self.X_train) * budget / 100)
        X_b = self.X_train.iloc[:n_rows]
        y_b = self.y_train[:n_rows]

        model = xgb.XGBClassifier(
            n_estimators=config['n_estimators'],
            max_depth=config['max_depth'],
            learning_rate=config['learning_rate'],
            subsample=config['subsample'],
            colsample_bytree=config['max_features'],  # max_features maps to colsample_bytree
            min_child_weight=config['min_child_weight'],
            use_label_encoder=False,
            eval_metric='logloss',
            random_state=42,
            n_jobs=1,
        )
        model.fit(X_b, y_b, verbose=False)

        y_proba = model.predict_proba(self.X_val)[:, 1]
        auc = roc_auc_score(self.y_val, y_proba)

        return {'loss': 1.0 - auc, 'info': {'auc': auc, 'n_rows': n_rows}}


# ConfigSpace: XGBoost hyperparameter search space
cs = CS.ConfigurationSpace(seed=42)
cs.add_hyperparameters([
    CSH.UniformIntegerHyperparameter('n_estimators',    lower=50,   upper=500, log=True),
    CSH.UniformIntegerHyperparameter('max_depth',       lower=3,    upper=10),
    CSH.UniformFloatHyperparameter('learning_rate',     lower=0.01, upper=0.3, log=True),
    CSH.UniformFloatHyperparameter('subsample',         lower=0.5,  upper=1.0),
    CSH.UniformFloatHyperparameter('max_features',      lower=0.3,  upper=1.0),  # colsample_bytree
    CSH.UniformIntegerHyperparameter('min_child_weight', lower=1,   upper=20),
])

# Start nameserver and run BOHB
NS = hpns.NameServer(run_id='xgb_hpo', host='127.0.0.1', port=None)
NS.start()

worker = XGBWorker(
    X_train=X_train, y_train=y_train,
    X_val=X_val,     y_val=y_val,
    run_id='xgb_hpo', nameserver='127.0.0.1',
)
worker.run(background=True)

result_logger = hpres.json_result_logger(
    directory='bohb_results', overwrite=True
)

# ← Problem: min_budget=10 means only 2,000 training rows
# At 2K rows, income and income_verified (r=0.94) have unstable relative importance
# BOHB multi-fidelity assumption requires ranking to be consistent across budgets
optimizer = BOHB(
    configspace=cs,
    run_id='xgb_hpo',
    nameserver='127.0.0.1',
    min_budget=10,     # ← 10% of 20K = 2,000 rows — TOO FEW for correlated features
    max_budget=100,    # ← 100% of 20K = 20,000 rows
    eta=3,             # fraction promoted at each level: 1/3 of configs advance
    result_logger=result_logger,
)

res = optimizer.run(n_iterations=50)
optimizer.shutdown(shutdown_workers=True)
NS.shutdown()

id2config = res.get_id2config_mapping()
incumbent = res.get_incumbent_id()
inc_config = id2config[incumbent]['config']

# Evaluate incumbent at full budget (max_budget=100)
n_rows_full = len(X_train)  # 16,000 rows (80% split)
model_inc = xgb.XGBClassifier(**{
    'n_estimators':    inc_config['n_estimators'],
    'max_depth':       inc_config['max_depth'],
    'learning_rate':   inc_config['learning_rate'],
    'subsample':       inc_config['subsample'],
    'colsample_bytree': inc_config['max_features'],
    'min_child_weight': inc_config['min_child_weight'],
    'use_label_encoder': False, 'eval_metric': 'logloss',
    'random_state': 42, 'n_jobs': -1,
})
model_inc.fit(X_train, y_train)
y_proba_inc = model_inc.predict_proba(X_val)[:, 1]
auc_bohb = roc_auc_score(y_val, y_proba_inc)
print(f"BOHB incumbent val_AUC: {auc_bohb:.4f}")
# BOHB incumbent val_AUC: 0.7401
# inc_config['max_features'] = 0.5  ← key: feature subsampling at 50%

# Grid search at max_budget=100 only (no multi-fidelity)
from sklearn.model_selection import ParameterGrid
best_auc_grid, best_grid_config = 0.0, None
for params in ParameterGrid({
    'n_estimators': [100, 200, 400],
    'max_depth': [4, 6, 8],
    'learning_rate': [0.05, 0.1, 0.2],
    'colsample_bytree': [0.6, 0.8, 1.0],
}):
    m = xgb.XGBClassifier(**params, subsample=0.8, min_child_weight=5,
                           use_label_encoder=False, eval_metric='logloss',
                           random_state=42, n_jobs=-1)
    m.fit(X_train, y_train)
    auc_g = roc_auc_score(y_val, m.predict_proba(X_val)[:, 1])
    if auc_g > best_auc_grid:
        best_auc_grid, best_grid_config = auc_g, params

print(f"Grid search val_AUC (max_budget=100 only): {best_auc_grid:.4f}")
# Grid search val_AUC (max_budget=100 only): 0.8101
# BOHB should outperform grid search — gap of 0.07 means multi-fidelity is broken

Why correlated feature pairs break BOHB’s multi-fidelity assumption at low min_budget

BOHB’s efficiency depends entirely on one assumption: configurations that perform well at low budget (few training rows) will also perform well at high budget (all training rows). If this assumption holds, BOHB can safely discard two-thirds (with eta=3) of configurations after evaluating them cheaply at min_budget, investing the saved compute in the promising third at higher budgets. If the assumption breaks — if good-at-low-budget and good-at-high-budget are uncorrelated — BOHB degrades to a random search with extra overhead, spending most evaluations at the wrong budget levels.

Correlated feature pairs are one of the most common sources of multi-fidelity assumption violations for tree-based models. When two features carry nearly identical information (Pearson r=0.94 for income and income_verified), XGBoost’s greedy split selection must arbitrarily choose one at each split. At small sample sizes (2,000 rows), statistical noise determines which of the correlated pair wins the first split: in this dataset, income (the raw numeric value) has slightly lower split variance at 2K rows and consistently wins around 73% of feature importance. At large sample sizes (20,000 rows), the data is rich enough to reveal that income_verified captures loan officer judgment beyond raw income — a subtle signal that only emerges when the model has seen enough examples to distinguish the two correlated features. At full budget, income_verified rises to 51% importance and income falls to 29%.

The configuration with max_features=0.5 (column subsampling at each tree) is especially affected. At min_budget=10 (2,000 rows), column subsampling at 0.5 stochastically includes or excludes the correlated pair at each tree. This configuration happened to produce trees that frequently sampled income (the dominant feature at 2K rows), giving it a good low-budget AUC. At max_budget=100 (20,000 rows), the same column subsampling now frequently excludes income_verified (the dominant feature at 20K rows), penalizing model quality. BOHB’s Bayesian surrogate model learned that max_features=0.5 is a good configuration, promoted it repeatedly, and built its acquisition function around the wrong signal. The Spearman rank correlation between configurations at min_budget and max_budget was 0.31 — barely better than random.

# Diagnostic: Spearman rank correlation between configs at min_budget vs max_budget
# hpbandster 0.7.x — audit multi-fidelity assumption validity

import numpy as np
import pandas as pd
from scipy.stats import spearmanr
import hpbandster.core.result as hpres

# Load completed BOHB run results
result = hpres.logged_results_to_HBS_result('bohb_results')
all_runs = result.get_all_runs()

# Extract AUC values for each config_id at min_budget=10 and max_budget=100
runs_at_min = {}  # config_id → AUC at budget=10
runs_at_max = {}  # config_id → AUC at budget=100

for run in all_runs:
    config_id = run.config_id
    budget    = run.budget
    if run.loss is None:
        continue  # skip failed runs
    auc = 1.0 - run.loss  # loss = 1 - AUC in our Worker

    if budget == 10:
        runs_at_min[config_id] = auc
    elif budget == 100:
        runs_at_max[config_id] = auc

# Find configs evaluated at BOTH budgets
shared_ids = set(runs_at_min.keys()) & set(runs_at_max.keys())
print(f"Configs evaluated at both min_budget=10 and max_budget=100: {len(shared_ids)}")
# Configs evaluated at both min_budget=10 and max_budget=100: 23

auc_min = [runs_at_min[cid] for cid in sorted(shared_ids)]
auc_max = [runs_at_max[cid] for cid in sorted(shared_ids)]

rho, p_value = spearmanr(auc_min, auc_max)
print(f"Spearman rank correlation (min_budget vs max_budget): rho={rho:.3f}, p={p_value:.4f}")
# Spearman rank correlation (min_budget vs max_budget): rho=0.310, p=0.1507
# rho=0.31: very low — budget=10 rankings are barely predictive of budget=100 rankings
# p=0.15: not even statistically significant — worse than random in practice

# Visualise the rank mismatch
df_ranks = pd.DataFrame({
    'config_id': sorted(shared_ids),
    'auc_at_min_budget': auc_min,
    'auc_at_max_budget': auc_max,
})
df_ranks['rank_at_min'] = df_ranks['auc_at_min_budget'].rank(ascending=False).astype(int)
df_ranks['rank_at_max'] = df_ranks['auc_at_max_budget'].rank(ascending=False).astype(int)
df_ranks['rank_delta']  = (df_ranks['rank_at_min'] - df_ranks['rank_at_max']).abs()

print("\nTop 5 worst-promoted configs (highest rank mismatch):")
print(df_ranks.nlargest(5, 'rank_delta')[
    ['config_id', 'auc_at_min_budget', 'auc_at_max_budget', 'rank_at_min', 'rank_at_max', 'rank_delta']
].to_string(index=False))
# config_id  auc_at_min_budget  auc_at_max_budget  rank_at_min  rank_at_max  rank_delta
# (0, 0, 2)              0.812              0.714            1           21          20  ← promoted!
# (0, 0, 7)              0.797              0.729            2           18          16
# (0, 0, 1)              0.779              0.753            3           15          12
# Config (0,0,2) = max_features=0.50: ranked 1st at budget=10, ranked 21st at budget=100
# BOHB built its surrogate model around this config's low-budget result

# Inspect the winning config's key hyperparameters
id2config = result.get_id2config_mapping()
promoted_config = id2config[(0, 0, 2)]['config']
print(f"\nPromoted config (rank 1 at min_budget, rank 21 at max_budget):")
for k, v in promoted_config.items():
    print(f"  {k}: {v}")
# max_features: 0.50  ← colsample_bytree=0.5; stochastic correlated-feature exclusion
# n_estimators: 287
# max_depth: 6
# learning_rate: 0.082
# subsample: 0.78
# min_child_weight: 4

# Feature importance audit: income vs income_verified at 2K vs 20K rows
import xgboost as xgb
from sklearn.model_selection import train_test_split

for n_rows, label in [(2000, 'budget=10 (2K rows)'), (20000, 'budget=100 (20K rows)')]:
    X_b = X_train.iloc[:n_rows]
    y_b = y_train[:n_rows]
    m = xgb.XGBClassifier(
        n_estimators=287, max_depth=6, learning_rate=0.082,
        subsample=0.78, colsample_bytree=1.0,  # use all features to isolate importance
        min_child_weight=4, use_label_encoder=False,
        eval_metric='logloss', random_state=42, n_jobs=-1,
    )
    m.fit(X_b, y_b)
    imp = pd.Series(m.feature_importances_, index=X_train.columns).sort_values(ascending=False)
    print(f"\nFeature importance at {label}:")
    print(f"  income:           {imp['income']:.3f}")
    print(f"  income_verified:  {imp['income_verified']:.3f}")

# Feature importance at budget=10 (2K rows):
#   income:           0.731
#   income_verified:  0.118
# Feature importance at budget=100 (20K rows):
#   income:           0.291
#   income_verified:  0.513
# → income and income_verified SWAP importance ranking between 2K and 20K rows
# → any config that subsamples features (max_features < 1.0) is unstable across budgets

Fix: increase min_budget from 10 to 50 so feature importance rankings stabilise

The fix is to raise min_budget from 10 to 50 — from 2,000 training rows to 10,000 training rows. At 10,000 rows, the dataset is large enough for XGBoost to consistently identify income_verified as the more informative feature of the correlated pair. The feature importance rankings at min_budget=50 align closely with those at max_budget=100: income_verified holds around 44% importance at 10K rows and 51% at 20K rows, with no rank swap. The Spearman rank correlation between configurations at the new min_budget and at max_budget rises from 0.31 to 0.87 — well within the range where BOHB’s multi-fidelity promotion is reliable.

With stable rankings, BOHB’s Bayesian surrogate model learns from meaningful low-budget outcomes. It stops promoting the max_features=0.5 configuration that was only good at 2,000 rows and instead focuses compute on configurations with higher colsample_bytree values that allow income_verified to participate in every tree. The incumbent found with min_budget=50 achieves val_AUC=0.80, an improvement of 0.06 AUC points over the broken min_budget=10 run. The 1.5-hour engagement recovered most of the 0.07-point gap versus grid search at max_budget=100.

# Fix: raise min_budget from 10 to 50 — ranking correlation 0.31→0.87
# hpbandster 0.7.x, Python 3.11

import numpy as np
import pandas as pd
from scipy.stats import spearmanr
from sklearn.metrics import roc_auc_score
import ConfigSpace as CS
import ConfigSpace.hyperparameters as CSH
import hpbandster.core.nameserver as hpns
import hpbandster.core.result as hpres
from hpbandster.optimizers import BOHB

# XGBWorker class is unchanged — only min_budget changes in the optimizer call

# Step 1: validate the new min_budget with a pilot run before committing 50 iterations
# Run 20 configs at min_budget=50 and max_budget=100, check Spearman rank correlation

NS_pilot = hpns.NameServer(run_id='xgb_pilot', host='127.0.0.1', port=None)
NS_pilot.start()

worker_pilot = XGBWorker(
    X_train=X_train, y_train=y_train,
    X_val=X_val,     y_val=y_val,
    run_id='xgb_pilot', nameserver='127.0.0.1',
)
worker_pilot.run(background=True)

result_logger_pilot = hpres.json_result_logger(
    directory='bohb_pilot', overwrite=True
)

# Pilot: single bracket from min_budget=50 to max_budget=100 with eta=2
# This generates configs evaluated at both 50 and 100, with minimal wasted compute
optimizer_pilot = BOHB(
    configspace=cs,
    run_id='xgb_pilot',
    nameserver='127.0.0.1',
    min_budget=50,   # ← NEW: 50% of 20K = 10,000 rows
    max_budget=100,  # ← unchanged
    eta=2,           # eta=2 for pilot: half of configs advance each level
    result_logger=result_logger_pilot,
)
res_pilot = optimizer_pilot.run(n_iterations=10)
optimizer_pilot.shutdown(shutdown_workers=True)
NS_pilot.shutdown()

# Compute Spearman rank correlation from pilot
pilot_runs = res_pilot.get_all_runs()
pilot_min, pilot_max = {}, {}
for run in pilot_runs:
    if run.loss is None:
        continue
    auc = 1.0 - run.loss
    if run.budget == 50:
        pilot_min[run.config_id] = auc
    elif run.budget == 100:
        pilot_max[run.config_id] = auc

shared_pilot = set(pilot_min.keys()) & set(pilot_max.keys())
print(f"Pilot: configs at both min=50 and max=100: {len(shared_pilot)}")
# Pilot: configs at both min=50 and max=100: 8

rho_new, p_new = spearmanr(
    [pilot_min[c] for c in sorted(shared_pilot)],
    [pilot_max[c] for c in sorted(shared_pilot)],
)
print(f"Pilot Spearman rank correlation (min=50 vs max=100): rho={rho_new:.3f}")
# Pilot Spearman rank correlation (min=50 vs max=100): rho=0.869
# rho=0.87: strong — min_budget=50 rankings are reliable predictors of max_budget=100

# Feature importance at new min_budget=50 (10,000 rows):
import xgboost as xgb
m_check = xgb.XGBClassifier(
    n_estimators=200, max_depth=6, learning_rate=0.1,
    subsample=0.8, colsample_bytree=1.0, min_child_weight=5,
    use_label_encoder=False, eval_metric='logloss', random_state=42, n_jobs=-1,
)
m_check.fit(X_train.iloc[:10000], y_train[:10000])
imp_check = pd.Series(m_check.feature_importances_, index=X_train.columns)
print(f"\nFeature importance at min_budget=50 (10K rows):")
print(f"  income:           {imp_check['income']:.3f}")
print(f"  income_verified:  {imp_check['income_verified']:.3f}")
# Feature importance at min_budget=50 (10K rows):
#   income:           0.338
#   income_verified:  0.441
# → income_verified already dominant at 10K rows — consistent with 20K ranking
# → correlated pair ranking is STABLE; multi-fidelity assumption holds

# Step 2: full BOHB run with min_budget=50
NS_fix = hpns.NameServer(run_id='xgb_fixed', host='127.0.0.1', port=None)
NS_fix.start()

worker_fix = XGBWorker(
    X_train=X_train, y_train=y_train,
    X_val=X_val,     y_val=y_val,
    run_id='xgb_fixed', nameserver='127.0.0.1',
)
worker_fix.run(background=True)

result_logger_fix = hpres.json_result_logger(
    directory='bohb_fixed', overwrite=True
)

optimizer_fix = BOHB(
    configspace=cs,
    run_id='xgb_fixed',
    nameserver='127.0.0.1',
    min_budget=50,   # ← FIXED: 50% of 20K = 10,000 rows training at lowest budget
    max_budget=100,
    eta=3,
    result_logger=result_logger_fix,
)

res_fix = optimizer_fix.run(n_iterations=50)
optimizer_fix.shutdown(shutdown_workers=True)
NS_fix.shutdown()

# Evaluate the fixed run's incumbent
id2config_fix = res_fix.get_id2config_mapping()
incumbent_fix = res_fix.get_incumbent_id()
inc_config_fix = id2config_fix[incumbent_fix]['config']

print(f"\nFixed incumbent config:")
for k, v in inc_config_fix.items():
    print(f"  {k}: {v}")
# max_features: 0.87  ← high colsample_bytree: income_verified participates in most trees
# n_estimators: 341
# max_depth: 5
# learning_rate: 0.067
# subsample: 0.82
# min_child_weight: 3

model_fix = xgb.XGBClassifier(**{
    'n_estimators':    inc_config_fix['n_estimators'],
    'max_depth':       inc_config_fix['max_depth'],
    'learning_rate':   inc_config_fix['learning_rate'],
    'subsample':       inc_config_fix['subsample'],
    'colsample_bytree': inc_config_fix['max_features'],
    'min_child_weight': inc_config_fix['min_child_weight'],
    'use_label_encoder': False, 'eval_metric': 'logloss',
    'random_state': 42, 'n_jobs': -1,
})
model_fix.fit(X_train, y_train)
y_proba_fix = model_fix.predict_proba(X_val)[:, 1]
auc_fix = roc_auc_score(y_val, y_proba_fix)
print(f"\nFixed BOHB val_AUC: {auc_fix:.4f}")
# Fixed BOHB val_AUC: 0.7998

# ---- After fix: summary ----
# BOHB (min_budget=10, Spearman rho=0.31): val_AUC = 0.74  ← multi-fidelity broken
# Grid search  (max_budget=100 only):       val_AUC = 0.81  ← BOHB was losing to grid
# BOHB (min_budget=50, Spearman rho=0.87): val_AUC = 0.80  ← multi-fidelity restored

# ---- Hour breakdown ----
# Triage:     0.5h — val_AUC=0.74 vs grid search 0.81; BOHB should outperform grid
#             search when multi-fidelity assumption holds; Spearman rank correlation
#             diagnostic run on existing results; rho=0.31 found
# Root cause: 0.5h — income/income_verified r=0.94; feature importance flip confirmed
#             at 2K vs 20K rows; max_features=0.5 config ranked 1st at budget=10,
#             21st at budget=100; min_budget=10 too few rows for stable rankings
# Fix:        0.25h — pilot run with min_budget=50; Spearman rho 0.31→0.87; full run
# Validation: 0.25h — val_AUC 0.74→0.80; 1.5h total

Related retainer posts

Track retainer hours without a client portal

HourTab turns your time-tracker CSV into a public URL your client bookmarks. No login. No portal. CSV in, URL out.

See pricing →