KerasTuner developer on retainer: stale Hyperband trial directory silently sets new hp.Float to None — l2_regularization never sampled — 50 wasted trials — 3 weeks — KerasTuner developer on monthly retainer

keras-tuner 1.4.x • Python 3.11 • TensorFlow 2.15.x • 50-trial Hyperband search • ResNet-50 CIFAR-10 fine-tuning • hp.Float('l2_regularization', 1e-5, 1e-2, sampling='log') added to model_builder() • my_dir/resnet_tuner/ not deleted before re-run • KerasTuner loads 50 stale trial JSONs missing l2_regularization key • hp.get('l2_regularization') returns None • tf.keras.regularizers.l2(None) → no regularization applied • val_accuracy plateau 0.78 unchanged over 50 new trials • developer checks TF version, ResNet weights, learning rate range — 3 weeks • fix: delete my_dir/resnet_tuner/ before re-running • l2_regularization correctly sampled • val_accuracy 0.78→0.84 • 1.5h retainer engagement

Northfield KerasTuner Search Ltd. ran a 50-trial Hyperband search over a ResNet-50 fine-tuning task on CIFAR-10 using keras-tuner 1.4.x and TensorFlow 2.15.x. After a first search that reached a best val_accuracy of 0.78, the developer added a new hp.Float('l2_regularization', 1e-5, 1e-2, sampling='log') hyperparameter to model_builder() and re-ran tuner.search() without deleting the existing my_dir/resnet_tuner/ directory. The second search completed 50 trials with no improvement in val_accuracy. The developer engaged a KerasTuner retainer developer after three weeks.

The model_builder() and tuner configuration at the time of the engagement:

# Northfield KerasTuner Search Ltd. — ResNet-50 Hyperband search
# keras-tuner 1.4.x, Python 3.11, TensorFlow 2.15.x

import keras_tuner as kt
import tensorflow as tf

# First search model_builder (before modification):
# def model_builder(hp):
#     num_dense = hp.Int('num_dense_units', 64, 512, step=64)
#     lr = hp.Float('learning_rate', 1e-5, 1e-2, sampling='log')
#     ...
#     Best val_accuracy: 0.78

# Second search model_builder (after adding l2_regularization):
def model_builder(hp):
    num_dense = hp.Int('num_dense_units', 64, 512, step=64)
    lr = hp.Float('learning_rate', 1e-5, 1e-2, sampling='log')
    l2_reg = hp.Float('l2_regularization', 1e-5, 1e-2, sampling='log')  # ← NEW

    base_model = tf.keras.applications.ResNet50(
        include_top=False, weights='imagenet', input_shape=(32, 32, 3)
    )
    base_model.trainable = True

    x = base_model.output
    x = tf.keras.layers.GlobalAveragePooling2D()(x)
    x = tf.keras.layers.Dense(
        num_dense,
        activation='relu',
        kernel_regularizer=tf.keras.regularizers.l2(l2_reg),  # ← l2_reg = None here
        # tf.keras.regularizers.l2(None) silently applies l2=0.0
    )(x)
    outputs = tf.keras.layers.Dense(10, activation='softmax')(x)

    model = tf.keras.Model(inputs=base_model.input, outputs=outputs)
    model.compile(
        optimizer=tf.keras.optimizers.Adam(learning_rate=lr),
        loss='sparse_categorical_crossentropy',
        metrics=['accuracy'],
    )
    return model

tuner = kt.Hyperband(
    model_builder,
    objective='val_accuracy',
    max_epochs=50,
    factor=3,
    directory='my_dir',         # ← stale directory still exists
    project_name='resnet_tuner' # ← stale trial JSONs still on disk
    # overwrite=False is the default
)

# Second search: 50 trials — l2_regularization silently None for all
tuner.search(x_train, y_train, epochs=50, validation_data=(x_val, y_val))

# After 50 new trials: best val_accuracy = 0.78 (no improvement)
# Developer conclusion: "l2_regularization has no effect" — incorrect
# Actual cause: l2_reg was None for every trial due to stale directory

Why the stale trial directory silences the new hyperparameter

KerasTuner persists every trial’s hyperparameter configuration as a JSON file at my_dir/resnet_tuner/trial_XXXX/trial.json. This JSON contains the full set of hyperparameter values for that trial, keyed by hyperparameter name. When tuner.search() is called with an existing directory and overwrite=False (the default), KerasTuner reloads the previous trial configurations for any trials it needs to restart or inherit from. The reloaded trial.json files were written during the first search, before l2_regularization existed. They do not contain a l2_regularization key.

When KerasTuner calls model_builder(hp) for a loaded trial, the internal HyperParameters object is populated from the JSON configuration. Any call to hp.Float('l2_regularization', ...) for a hyperparameter not present in the trial’s JSON returns None rather than raising an error — KerasTuner treats missing keys as optional overrides, defaulting to None. The None value then flows into tf.keras.regularizers.l2(None), which TensorFlow interprets as a regularization strength of 0.0 (equivalent to no regularization). No warning is emitted. All 50 new trials run with l2_regularization effectively disabled, making the second search identical to the first in terms of the regularization axis.

# Diagnostic: check hp.values inside model_builder() to detect None

import keras_tuner as kt
import json, os

# Inspect a stale trial JSON — check for missing l2_regularization key:
trial_dir = 'my_dir/resnet_tuner/trial_0000/'
trial_json_path = os.path.join(trial_dir, 'trial.json')

with open(trial_json_path) as f:
    trial_data = json.load(f)

hp_values = trial_data.get('hyperparameters', {}).get('values', {})
print(hp_values)
# Output: {'num_dense_units': 192, 'learning_rate': 0.00031}
# Note: 'l2_regularization' key is ABSENT — was not present in first search

# Add a debug print inside model_builder() to catch None at runtime:
def model_builder_debug(hp):
    num_dense = hp.Int('num_dense_units', 64, 512, step=64)
    lr = hp.Float('learning_rate', 1e-5, 1e-2, sampling='log')
    l2_reg = hp.Float('l2_regularization', 1e-5, 1e-2, sampling='log')

    print(f"DEBUG hp.values: {hp.values}")
    # Output during second search (stale trials):
    # DEBUG hp.values: {'num_dense_units': 192, 'learning_rate': 0.00031,
    #                   'l2_regularization': None}
    # ← None confirms stale directory is silencing the new hyperparameter

    # Demonstrate TF's silent None handling:
    import tensorflow as tf
    reg = tf.keras.regularizers.l2(None)
    print(f"l2_reg with None argument: l2={reg.l2}")  # Output: l2=0.0
    # TF treats None as 0.0 — no warning, no error
    ...

The fix: delete the trial directory or use overwrite=True

The fix is to clear the stale trial directory before re-running the search after any modification to model_builder(). This can be done by passing overwrite=True to the tuner constructor, which causes KerasTuner to delete and recreate the project directory at the start of tuner.search(). Alternatively, manually deleting my_dir/resnet_tuner/ before instantiating the tuner achieves the same result. Once the directory is clean, KerasTuner starts all trials from scratch with fresh HyperParameters objects, and hp.Float('l2_regularization', ...) is correctly sampled from its specified range.

# Fix: use overwrite=True when model_builder() has changed
# keras-tuner 1.4.x, Northfield KerasTuner Search Ltd.

import keras_tuner as kt
import tensorflow as tf

def model_builder_fixed(hp):
    num_dense = hp.Int('num_dense_units', 64, 512, step=64)
    lr = hp.Float('learning_rate', 1e-5, 1e-2, sampling='log')
    l2_reg = hp.Float('l2_regularization', 1e-5, 1e-2, sampling='log')

    # Optional guard against future None regressions:
    if l2_reg is None:
        raise ValueError(
            f"l2_regularization is None — stale trial directory? "
            f"Use overwrite=True or delete my_dir/resnet_tuner/. "
            f"hp.values={hp.values}"
        )

    base_model = tf.keras.applications.ResNet50(
        include_top=False, weights='imagenet', input_shape=(32, 32, 3)
    )
    base_model.trainable = True

    x = base_model.output
    x = tf.keras.layers.GlobalAveragePooling2D()(x)
    x = tf.keras.layers.Dense(
        num_dense,
        activation='relu',
        kernel_regularizer=tf.keras.regularizers.l2(l2_reg),
    )(x)
    outputs = tf.keras.layers.Dense(10, activation='softmax')(x)

    model = tf.keras.Model(inputs=base_model.input, outputs=outputs)
    model.compile(
        optimizer=tf.keras.optimizers.Adam(learning_rate=lr),
        loss='sparse_categorical_crossentropy',
        metrics=['accuracy'],
    )
    return model

tuner = kt.Hyperband(
    model_builder_fixed,
    objective='val_accuracy',
    max_epochs=50,
    factor=3,
    directory='my_dir',
    project_name='resnet_tuner',
    overwrite=True,   # ← clears my_dir/resnet_tuner/ before search
                      # Required whenever model_builder() hyperparameters change
)

tuner.search(
    x_train, y_train,
    epochs=50,
    validation_data=(x_val, y_val),
    callbacks=[tf.keras.callbacks.EarlyStopping(patience=5)],
)

best_hps = tuner.get_best_hyperparameters(num_trials=1)[0]
print(f"Best num_dense_units:   {best_hps.get('num_dense_units')}")
print(f"Best learning_rate:     {best_hps.get('learning_rate'):.5f}")
print(f"Best l2_regularization: {best_hps.get('l2_regularization'):.6f}")
print(f"Best val_accuracy:      {tuner.oracle.get_best_trials(1)[0].score:.4f}")

# Output after fix:
# Best num_dense_units:   256
# Best learning_rate:     0.00028
# Best l2_regularization: 0.000041
# Best val_accuracy:      0.8411

# Before fix (stale dir, l2_reg=None): best val_accuracy = 0.7800
# After fix  (overwrite=True):         best val_accuracy = 0.8411

Validation

# Validation log — November 2026
# keras-tuner 1.4.x, Python 3.11, TensorFlow 2.15.x
# 50-trial Hyperband search over ResNet-50 fine-tuning (CIFAR-10)

# ---- Before fix: stale my_dir/resnet_tuner/ directory ----
# hp.values['l2_regularization'] for all 50 trials: None
# tf.keras.regularizers.l2(None): l2=0.0 (no regularization)
# Best val_accuracy after 50 new trials: 0.7800 (same as first search)
# l2_regularization had zero effect on any trial

# ---- After fix: overwrite=True ----
# hp.values['l2_regularization'] for all 50 trials: sampled in [1e-5, 1e-2]
# l2_regularization range explored: 0.000011 → 0.009400
# Best val_accuracy after 50 new trials: 0.8411
# Best trial: num_dense_units=256, learning_rate=0.00028,
#             l2_regularization=0.000041

# ---- Improvement ----
# val_accuracy: 0.7800 → 0.8411 (+7.8%)
# l2_regularization correctly explored: [1e-5, 1e-2] log-uniform

# ---- Hour breakdown ----
# Triage:     0.5h — val_accuracy plateau; DEBUG print shows
#             hp.values['l2_regularization']=None for all trials;
#             stale my_dir/resnet_tuner/ confirmed
# Root cause: 0.25h — trial JSONs from first search missing l2_reg key;
#             hp.get() returns None; TF regularizers.l2(None) → l2=0.0
# Fix:        0.25h — added overwrite=True to Hyperband constructor;
#             added None guard in model_builder_fixed()
# Validation: 0.5h — 50 trials; l2_reg sampled correctly;
#             val_accuracy 0.84; 1.5h total

Related retainer posts

Track retainer hours without a client portal

HourTab turns your time-tracker CSV into a public URL your client bookmarks. No login. No portal. CSV in, URL out.

See pricing →