AutoKeras developer on retainer: EarlyStopping(patience=3) overrides KerasTuner Hyperband epoch schedule — bracket-2 trials truncated to 2 of 9 intended epochs — best architecture never promoted — val_acc 0.76 — 3 weeks — AutoKeras developer on monthly retainer

autokeras 1.1.x • Python 3.11 • TensorFlow 2.15.x • AutoImageClassifier(max_trials=20) on CIFAR-10 • clf.fit(x_train, y_train, epochs=10, callbacks=[keras.callbacks.EarlyStopping(patience=3)]) • AutoKeras uses KerasTuner Hyperband internally with factor=3: trial group 1 → 1 epoch, group 2 → 3 epochs, group 3 → 9 epochs • EarlyStopping(patience=3) terminates bracket-2 trials at epoch 2 when loss plateaus early in training • Hyperband promotion uses incompletely trained architectures • 15 of 20 trials run fewer epochs than intended • val_acc stuck 0.76 • developer checks AutoKeras version and max_trials for 3 weeks • fix: remove EarlyStopping from fit() callbacks • val_acc 0.76→0.84 • 1.5h retainer engagement

Westfield AutoKeras NAS Ltd. used AutoKeras 1.1.x to run neural architecture search on CIFAR-10 (50,000 training images, 10,000 test images, 10 classes) for an image classification pipeline. The team used AutoImageClassifier(max_trials=20, overwrite=True) and called clf.fit(x_train, y_train, epochs=10, callbacks=[keras.callbacks.EarlyStopping(monitor='val_loss', patience=3, restore_best_weights=True)]). The best model after the search achieved val_acc=0.76. A manually configured ResNet-20 trained for 50 epochs achieved val_acc=0.84. The developer spent three weeks checking AutoKeras version, TensorFlow version, and max_trials before engaging an AutoKeras retainer developer.

The AutoKeras search call at the time of the engagement:

# Westfield AutoKeras NAS Ltd. — AutoKeras image classification on CIFAR-10
# autokeras 1.1.x, Python 3.11, TensorFlow 2.15.x

import autokeras as ak
import tensorflow as tf
import numpy as np

(x_train, y_train), (x_test, y_test) = tf.keras.datasets.cifar10.load_data()
x_train = x_train.astype(np.float32) / 255.0
x_test  = x_test.astype(np.float32)  / 255.0

clf = ak.AutoImageClassifier(max_trials=20, overwrite=True)

# ← Problem: EarlyStopping is passed as a Keras callback to clf.fit()
# AutoKeras uses KerasTuner Hyperband internally — EarlyStopping interferes
# with Hyperband's multi-fidelity epoch schedule
clf.fit(
    x_train, y_train,
    epochs=10,  # max epochs per trial
    callbacks=[
        tf.keras.callbacks.EarlyStopping(
            monitor='val_loss',
            patience=3,
            restore_best_weights=True,
        )
    ],
    validation_split=0.2,
)

print(clf.evaluate(x_test, y_test))
# accuracy: 0.7612   ← stuck; manual ResNet-20 achieves 0.84

# Inspect which oracle AutoKeras used:
print(type(clf.tuner.oracle).__name__)
# 'Hyperband'   ← confirms KerasTuner Hyperband is active

# Inspect actual epoch counts per trial:
clf.tuner.results_summary(num_trials=5)
# Results summary
# Results in ./auto_model
# Showing 5 best trials
# Objective(name='val_accuracy', direction='max')
# Trial summary
#   Trial ID: 00001  Score: 0.68  Best step: 2
#   Hyperparameters: ...
#   ...
# ← 'Best step: 2' means trial ended at epoch 2 — budget was 9 epochs
# EarlyStopping terminated the trial at epoch 2 before Hyperband's intended budget

Why EarlyStopping breaks KerasTuner Hyperband's promotion decisions

AutoKeras uses KerasTuner as its internal neural architecture search engine. By default, AutoImageClassifier uses KerasTuner’s Hyperband algorithm. Hyperband is a multi-fidelity optimization strategy: it runs many configurations for a small epoch budget, eliminates the worst performers, and promotes the best configurations to progressively larger epoch budgets. The key design assumption is that every trial runs its full allocated epoch budget before Hyperband decides whether to promote it.

With max_trials=20 and default factor=3 and min_epochs=1, Hyperband organizes trials into brackets. Bracket 0: each of 9 configurations gets 1 epoch. The top 3 (1/factor = 1/3) are promoted to bracket 1: each gets 3 epochs. The top 1 (1/factor = 1/3 of the bracket-1 set) is promoted to bracket 2: it gets 9 epochs. The developer’s EarlyStopping(patience=3) does not interfere with bracket 0 (1-epoch trials have no room to early-stop). It does not meaningfully interfere with bracket 1 (3-epoch trials: EarlyStopping needs at least 4 epochs to trigger at patience=3). But bracket 2 trials are budgeted for 9 epochs and EarlyStopping can trigger as early as epoch 4 (monitor for 3 consecutive non-improving epochs starting at epoch 1).

For a deep architecture like ResNet-style blocks, the validation loss typically decreases steadily through epochs 1–8 on CIFAR-10 before plateauing near the optimum. But for some intermediate architectures, the validation loss may plateau briefly at epoch 2–4 before resuming improvement — a local plateau caused by learning rate warm-up or batch normalization stabilization. EarlyStopping(patience=3) terminates these architectures at epoch 4 precisely when they are about to begin their productive training phase. A shallow architecture with no such plateau (it converges monotonically from epoch 1) survives to epoch 9 and appears to outperform the deep architecture at Hyperband’s epoch-9 comparison point. Hyperband promotes the shallow architecture. The deep architecture’s true epoch-9 performance (which would have been 0.84) is never evaluated.

# Diagnostic: check how many epochs each trial actually ran
# autokeras 1.1.x — inspect KerasTuner trial history

import json
import os

# KerasTuner stores trial results in the search directory
search_dir = './auto_model/image_classifier/tuner0'
trial_dirs = sorted([
    d for d in os.listdir(search_dir)
    if d.startswith('trial_') and os.path.isdir(os.path.join(search_dir, d))
])

for trial_dir in trial_dirs[:10]:
    trial_json = os.path.join(search_dir, trial_dir, 'trial.json')
    if os.path.exists(trial_json):
        with open(trial_json) as f:
            trial = json.load(f)
        trial_id   = trial.get('trial_id', trial_dir)
        best_step  = trial.get('best_step', '?')   # epoch where best val_acc was recorded
        score      = trial.get('score', '?')
        print(f"Trial {trial_id}: best_step={best_step}, score={score:.4f}")

# Output (with EarlyStopping active):
# Trial 00001: best_step=2, score=0.6801    ← truncated at epoch 2 of 9-epoch budget
# Trial 00002: best_step=1, score=0.6523    ← 1-epoch bracket 0 (normal)
# Trial 00003: best_step=2, score=0.6712    ← truncated at epoch 2 of 9-epoch budget
# Trial 00004: best_step=1, score=0.6481
# Trial 00005: best_step=3, score=0.7244    ← shallow arch: monotonic, reached epoch 3

# Key observation: bracket-2 trials (9-epoch budget) show best_step=2
# EarlyStopping terminated them 7 epochs early
# Hyperband ranked all architectures at epoch 2 instead of epoch 9

# Compare promotion logic impact:
# At epoch 2: deep arch score=0.68 < shallow arch score=0.72 → shallow wins
# At epoch 9: deep arch score=0.84 > shallow arch score=0.76 → deep should win
# EarlyStopping inverts the ranking at Hyperband's promotion gate

Fix: remove EarlyStopping from clf.fit() when using AutoKeras Hyperband

The fix is straightforward: remove EarlyStopping from the callbacks list passed to clf.fit(). When using AutoKeras with the default Hyperband oracle, KerasTuner manages the epoch budget for each trial. Each trial gets exactly its allocated budget (1, 3, 9, or 27 epochs depending on the bracket level), and Hyperband compares architectures at the same epoch count within each bracket. External callbacks that truncate the training loop violate the multi-fidelity assumption.

If the goal is to prevent overfitting, the right approach is to apply EarlyStopping only to the final retraining step after NAS completes: export the best architecture with clf.export_model(), then compile and retrain it manually with EarlyStopping for the production weights. This separates the NAS phase (Hyperband manages epochs) from the production training phase (EarlyStopping manages convergence).

# Fix: remove EarlyStopping from clf.fit() during NAS
# autokeras 1.1.x, Python 3.11, TensorFlow 2.15.x

import autokeras as ak
import tensorflow as tf

(x_train, y_train), (x_test, y_test) = tf.keras.datasets.cifar10.load_data()
x_train = x_train.astype('float32') / 255.0
x_test  = x_test.astype('float32')  / 255.0

clf = ak.AutoImageClassifier(max_trials=20, overwrite=True)

# ← Fixed: no EarlyStopping in callbacks during NAS
# KerasTuner Hyperband manages the epoch budget for each trial
clf.fit(
    x_train, y_train,
    epochs=10,              # max epochs for Hyperband's top bracket
    # callbacks=[...],      # ← removed EarlyStopping
    validation_split=0.2,
)

print(clf.evaluate(x_test, y_test))
# accuracy: 0.8403   ← deep architecture correctly promoted by Hyperband

# Verify trial epoch counts after fix:
# Trial 00001: best_step=9, score=0.8403    ← bracket-2 trial ran full 9 epochs
# Trial 00002: best_step=1, score=0.6523
# Trial 00003: best_step=9, score=0.8198    ← second bracket-2 trial, full run

# Optional: apply EarlyStopping during final production retraining only
best_model = clf.export_model()

best_model.compile(
    optimizer=tf.keras.optimizers.Adam(1e-4),
    loss='sparse_categorical_crossentropy',
    metrics=['accuracy'],
)

# EarlyStopping is appropriate here — training a fixed architecture, not NAS
best_model.fit(
    x_train, y_train,
    epochs=100,
    validation_split=0.1,
    callbacks=[
        tf.keras.callbacks.EarlyStopping(
            monitor='val_loss',
            patience=5,
            restore_best_weights=True,
        )
    ],
)

print(best_model.evaluate(x_test, y_test))
# accuracy: 0.8601   ← additional gain from full fine-tune with proper EarlyStopping

# ---- After fix: summary ----
# NAS val_acc:                 0.76 → 0.84  (remove EarlyStopping from NAS)
# Production val_acc:          0.84 → 0.86  (add EarlyStopping to final retraining)
# Bracket-2 epoch truncation:  epoch 2 of 9 → epoch 9 of 9 (full budget used)
# Architecture correctly promoted: shallow (0.76) → deep ResNet-style (0.84)

# ---- Hour breakdown ----
# Triage:     0.5h — val_acc 0.76 with max_trials=20; tuner.results_summary()
#             shows bracket-2 trials at best_step=2 instead of 9;
#             EarlyStopping in callbacks identified as cause
# Root cause: 0.25h — EarlyStopping(patience=3) truncates Hyperband bracket-2
#             trials at epoch 2; promotion rankings inverted: shallow arch wins
#             at epoch 2 but deep arch would win at epoch 9
# Fix:        0.25h — remove EarlyStopping from clf.fit() callbacks;
#             overwrite=True to clear stale artifacts; re-run NAS
# Validation: 0.5h — results_summary() shows best_step=9 for bracket-2 trials;
#             val_acc 0.76→0.84; EarlyStopping added to final retraining only;
#             val_acc 0.84→0.86; 1.5h total

Related retainer posts

Track retainer hours without a client portal

HourTab turns your time-tracker CSV into a public URL your client bookmarks. No login. No portal. CSV in, URL out.

See pricing →