SMAC3 developer on retainer: identical data across all 5 instances makes surrogate model ignore configuration space — incumbent worse than default RF after 200 trials — 3 weeks — SMAC3 developer on monthly retainer

SMAC3 2.0.x • Python 3.11 • ConfigSpace 1.1.x • scikit-learn 1.4.x • instance_features=np.array([[1],[2],[3],[4],[5]]) • all 5 instances call pd.read_csv('fraud_50k.csv') with random_state=42 • identical 40K/10K splits across all instances • instance performance variance std < 0.0001 • RandomForestSurrogate response surface flat • EI acquisition degenerate • best val_AUC 0.71 after 200 trials — worse than default RF (0.72) • developer checks SMAC3 version, ConfigSpace, Python version — 3 weeks • fix: random_state=instance_id in train_test_split • instance variance std 0.000→0.018 • best val_AUC 0.71→0.78 • 1.5h retainer engagement

Northfield SMAC BO Ltd. used SMAC3 2.0.x to optimize a scikit-learn Random Forest classifier on a fraud detection dataset, using five instances with instance_features=np.array([[1],[2],[3],[4],[5]]) to enable multi-instance generalization. After 200 trials, the incumbent val_AUC was 0.71 — slightly worse than a default scikit-learn RandomForestClassifier at 0.72 and far below the team’s manual Optuna baseline of 0.79. The developer spent three weeks checking SMAC3 version, ConfigSpace definition, Python version, and scikit-learn version before engaging a SMAC3 retainer developer.

The SMAC3 scenario and target function at the time of the engagement:

# Northfield SMAC BO Ltd. — SMAC3 multi-instance HPO
# SMAC3 2.0.x, Python 3.11, ConfigSpace 1.1.x, scikit-learn 1.4.x

import numpy as np
import pandas as pd
from ConfigSpace import ConfigurationSpace, Integer, Float
from smac import Scenario
from smac import HyperparameterOptimizationFacade as HPOFacade
from smac import RunHistory
from sklearn.ensemble import RandomForestClassifier
from sklearn.model_selection import train_test_split
from sklearn.metrics import roc_auc_score

# ConfigurationSpace: Random Forest n_estimators and max_features
cs = ConfigurationSpace(seed=42)
cs.add([
    Integer('n_estimators',    (100, 500)),
    Float('max_features',      (0.1, 1.0)),
    Integer('min_samples_leaf', (1, 20)),
])

# 5 instances — numeric instance features for SMAC surrogate
instance_features = {
    '0': [1], '1': [2], '2': [3], '3': [4], '4': [5]
}

scenario = Scenario(
    configspace=cs,
    instances=list(instance_features.keys()),
    instance_features=instance_features,
    n_trials=200,
    seed=42,
)

def load_instance(instance_id: str):
    # ← Problem: same CSV, same random_state for all 5 instances
    df = pd.read_csv('fraud_50k.csv')
    X = df.drop('label', axis=1).values
    y = df['label'].values
    # Every instance uses the same random_state=42 → identical split
    return train_test_split(X, y, test_size=0.2, random_state=42)

def target_function(config, instance: str, seed: int) -> float:
    X_train, X_test, y_train, y_test = load_instance(instance)
    clf = RandomForestClassifier(
        n_estimators=config['n_estimators'],
        max_features=config['max_features'],
        min_samples_leaf=config['min_samples_leaf'],
        random_state=seed,
        n_jobs=4,
    )
    clf.fit(X_train, y_train)
    y_prob = clf.predict_proba(X_test)[:, 1]
    return 1.0 - roc_auc_score(y_test, y_prob)   # minimize

# ← Problem: all 5 instances load identical data
# Instance 0: fraud_50k.csv, random_state=42 → split A (40K/10K)
# Instance 1: fraud_50k.csv, random_state=42 → split A (40K/10K)
# ...
# Instance 4: fraud_50k.csv, random_state=42 → split A (40K/10K)
# For any configuration c: target_function(c, '0') ≈ target_function(c, '4')
# Instance variance: std across instances ≈ 0.0001 (numerical noise only)
# SMAC3 surrogate sees no instance-specific variance
# → response surface is flat across all config dimensions
# → EI acquisition returns near-zero values everywhere
# → incumbent stays at first randomly-sampled configuration

smac = HPOFacade(scenario, target_function)
incumbent = smac.optimize()

print(f"Best val_AUC: {1 - smac.validate(incumbent):.4f}")
# Output: Best val_AUC: 0.7100
# Default RF (n_estimators=100, max_features='sqrt'): 0.7200
# SMAC3 incumbent is worse than the default — 200 trials wasted

Why identical instance data makes the SMAC3 surrogate useless

SMAC3’s multi-instance optimization is designed around the assumption that evaluating the same configuration on multiple instances reveals how well it generalizes. Instance variance — the variation in a configuration’s performance across different instances — is the signal the surrogate model uses to separate configurations that work well only on specific data distributions from those that generalize broadly. When all instances contain the same data, this signal disappears entirely.

Internally, SMAC3’s RandomForestSurrogate takes as input a feature vector that concatenates the configuration’s hyperparameter values with the numeric instance features (from instance_features). The surrogate is trained on the runhistory — all (configuration, instance, cost) triples observed so far. When every (config_A, instance_0) through (config_A, instance_4) produces the same cost (because the instances are identical), the surrogate’s Random Forest sees no variance to explain. The variance across the five instance feature vectors [1], [2], [3], [4], [5] is non-zero (they differ numerically), but the response (cost) is constant across those feature vectors for any given configuration. The surrogate learns that neither configuration hyperparameters nor instance features predict cost variation — every prediction is essentially the mean of all observed costs.

With a flat surrogate, the Expected Improvement acquisition function computes near-zero EI everywhere: since the surrogate predicts the same cost for every unexplored point and that prediction equals the current incumbent cost, the probability of improvement is zero. SMAC3 falls back to random exploration with its intensifier, but the random configurations evaluated are still all measured as nearly identical to the incumbent, reinforcing the flat surface. After 200 trials, the incumbent is whatever configuration happened to score slightly best due to random fluctuation — not better than the scikit-learn default.

# Diagnostic: check instance variance in runhistory
# SMAC3 2.0.x, Python 3.11

from smac.runhistory import TrialKey, TrialValue
import numpy as np
from collections import defaultdict

# After smac.optimize() completes, audit instance variance per configuration:
runhistory = smac.runhistory
all_runs = runhistory.get_all_configs()

# For each configuration, collect cost per instance:
config_instance_costs = defaultdict(dict)
for trial_key, trial_value in runhistory.items():
    config = runhistory.get_config(trial_key.config_id)
    config_key = str(sorted(config.get_dictionary().items()))
    instance = trial_key.instance
    config_instance_costs[config_key][instance] = trial_value.cost

# Check variance across instances for first 5 configs:
print("Instance cost variance per configuration:")
for i, (cfg_key, instance_costs) in enumerate(list(config_instance_costs.items())[:5]):
    costs = list(instance_costs.values())
    if len(costs) > 1:
        variance = np.std(costs)
        print(f"  Config {i}: std across {len(costs)} instances = {variance:.6f}")
# Output:
#   Config 0: std across 5 instances = 0.000031   ← effectively zero
#   Config 1: std across 5 instances = 0.000028
#   Config 2: std across 5 instances = 0.000019
#   Config 3: std across 5 instances = 0.000041
#   Config 4: std across 5 instances = 0.000025
# All variances under 0.0001 → instances are providing zero signal

# Verify data identity across instances:
def verify_instance_data_identity():
    splits = []
    for instance_id in range(5):
        df = pd.read_csv('fraud_50k.csv')
        X = df.drop('label', axis=1).values
        y = df['label'].values
        X_train, X_test, y_train, y_test = train_test_split(
            X, y, test_size=0.2, random_state=42   # same seed for all
        )
        splits.append(X_train[:5, 0].tolist())   # first 5 rows, first feature

    for i in range(1, 5):
        identical = splits[i] == splits[0]
        print(f"  Instance {i} data identical to instance 0: {identical}")
    # Output:
    #   Instance 1 data identical to instance 0: True
    #   Instance 2 data identical to instance 0: True
    #   Instance 3 data identical to instance 0: True
    #   Instance 4 data identical to instance 0: True

verify_instance_data_identity()

The fix: use random_state=instance_id in train_test_split per instance

The fix is to use the integer instance ID as the random_state in train_test_split within load_instance(), so each of the five instances receives a genuinely different 40K/10K data split. With different splits, some configurations overfit to specific data distributions while others generalize broadly — creating real instance variance for the SMAC3 surrogate to model. The surrogate can then learn which regions of the configuration space produce robust performance across diverse splits, and the EI acquisition function can meaningfully prioritize those regions.

# Fix: random_state=instance_id for unique data splits per instance
# SMAC3 2.0.x, Northfield SMAC BO Ltd.

import numpy as np
import pandas as pd
from ConfigSpace import ConfigurationSpace, Integer, Float
from smac import Scenario
from smac import HyperparameterOptimizationFacade as HPOFacade
from sklearn.ensemble import RandomForestClassifier
from sklearn.model_selection import train_test_split
from sklearn.metrics import roc_auc_score

cs = ConfigurationSpace(seed=42)
cs.add([
    Integer('n_estimators',    (100, 500)),
    Float('max_features',      (0.1, 1.0)),
    Integer('min_samples_leaf', (1, 20)),
])

instance_features = {
    '0': [1], '1': [2], '2': [3], '3': [4], '4': [5]
}

scenario = Scenario(
    configspace=cs,
    instances=list(instance_features.keys()),
    instance_features=instance_features,
    n_trials=200,
    seed=42,
)

def load_instance_fixed(instance_id: str):
    # ← Fix: random_state=int(instance_id) for unique split per instance
    df = pd.read_csv('fraud_50k.csv')
    X = df.drop('label', axis=1).values
    y = df['label'].values
    return train_test_split(
        X, y,
        test_size=0.2,
        random_state=int(instance_id),   # 0, 1, 2, 3, 4 → different splits
        shuffle=True,
    )

def target_function_fixed(config, instance: str, seed: int) -> float:
    X_train, X_test, y_train, y_test = load_instance_fixed(instance)
    clf = RandomForestClassifier(
        n_estimators=config['n_estimators'],
        max_features=config['max_features'],
        min_samples_leaf=config['min_samples_leaf'],
        random_state=seed,
        n_jobs=4,
    )
    clf.fit(X_train, y_train)
    y_prob = clf.predict_proba(X_test)[:, 1]
    return 1.0 - roc_auc_score(y_test, y_prob)

smac_fixed = HPOFacade(scenario, target_function_fixed)
incumbent_fixed = smac_fixed.optimize()

# Check instance variance after fix:
runhistory_fixed = smac_fixed.runhistory
from collections import defaultdict
config_instance_costs_fixed = defaultdict(dict)
for trial_key, trial_value in runhistory_fixed.items():
    config = runhistory_fixed.get_config(trial_key.config_id)
    config_key = str(sorted(config.get_dictionary().items()))
    config_instance_costs_fixed[config_key][trial_key.instance] = trial_value.cost

print("Instance cost variance per configuration (after fix):")
for i, (cfg_key, instance_costs) in enumerate(list(config_instance_costs_fixed.items())[:5]):
    costs = list(instance_costs.values())
    if len(costs) > 1:
        variance = np.std(costs)
        print(f"  Config {i}: std across {len(costs)} instances = {variance:.6f}")
# Output:
#   Config 0: std across 5 instances = 0.018432   ← genuine variance
#   Config 1: std across 5 instances = 0.021107
#   Config 2: std across 5 instances = 0.014889
#   Config 3: std across 5 instances = 0.019540
#   Config 4: std across 5 instances = 0.016774
# All variances above 0.01 → surrogate can model generalization

best_cost = smac_fixed.validate(incumbent_fixed)
print(f"Best val_AUC (fixed): {1 - best_cost:.4f}")
# Output: Best val_AUC (fixed): 0.7812

print(f"Best config:")
print(f"  n_estimators    = {incumbent_fixed['n_estimators']}")
print(f"  max_features    = {incumbent_fixed['max_features']:.4f}")
print(f"  min_samples_leaf = {incumbent_fixed['min_samples_leaf']}")
# Output:
#   n_estimators    = 387
#   max_features    = 0.4231
#   min_samples_leaf = 3

# Before fix: instance variance ~0.00003; best val_AUC = 0.71 (200 trials wasted)
# After fix:  instance variance ~0.018;   best val_AUC = 0.78

Validation

# Validation log — November 2026
# SMAC3 2.0.x, Python 3.11, ConfigSpace 1.1.x, scikit-learn 1.4.x
# Multi-instance HPO: Random Forest on 50K fraud dataset, 5 instances

# ---- Before fix: random_state=42 for all 5 instances ----
# Instance variance (std per config):  ~0.00003 (effectively zero)
# SMAC surrogate:                      flat response surface
# EI acquisition:                      near-zero everywhere
# Trials exploring new configs:        ~20% (rest: intensifier re-evaluating incumbent)
# Best val_AUC after 200 trials:       0.710
# Default RF baseline:                 0.720   (SMAC3 did worse)

# ---- After fix: random_state=instance_id (0, 1, 2, 3, 4) ----
# Instance variance (std per config):  ~0.018 (meaningful variance)
# SMAC surrogate:                      meaningful response surface
# EI acquisition:                      explores generalization-robust configs
# Trials exploring new configs:        ~72%
# Best val_AUC after 200 trials:       0.781

# ---- Improvement ----
# val_AUC: 0.710 → 0.781 (+10.0%)
# Incumbent n_estimators: 387, max_features: 0.42, min_samples_leaf: 3

# ---- Comparison ----
# Default RF (sklearn defaults):       0.720
# Manual Optuna baseline (200 trials): 0.790
# SMAC3 fixed (200 trials, 5 instances): 0.781
# Note: SMAC3 near-matches Optuna baseline with multi-instance generalization

# ---- Hour breakdown ----
# Triage:     0.5h — incumbent val_AUC 0.71; instance variance std ~0.00003;
#             all 5 load_instance() calls use random_state=42
# Root cause: 0.25h — identical splits → zero instance variance →
#             flat surrogate → EI degenerate
# Fix:        0.25h — random_state=int(instance_id) in train_test_split
# Validation: 0.5h — instance variance 0.018; best val_AUC 0.781; 1.5h total

Related retainer posts

• scikit-optimize developer on retainer — near-zero GP noise from initial_y std makes EI degenerate — same point returned 80 times — 3 weeks
• Optuna developer on retainer — study.best_trial full table scan on trial_intermediate_values — 32 s after 200 trials — 3 weeks
• FLAML developer on retainer — 5-fold CV on 500K rows allows only 2 trials in 600s budget — 3 weeks