Hyperopt developer on retainer: Keras model weights stored in MongoTrials result dict — BSON document 16.8 MB — MongoDB 16 MB limit — BSONError at trial 150 — 3 weeks — Hyperopt developer on monthly retainer

hyperopt 0.2.7 • Python 3.11 • MongoDB 6.0 • MongoTrials • 300-trial TPE LSTM search • model.get_weights() in objective return dict • 14 float32 arrays × 15.8 MB total • bson.encode(result_dict) = 16,793,616 bytes • MongoDB 16 MB BSON document limit • bson.errors.BSONError: BSON document too large at trial 150 • trials 1–149 succeed (document sizes vary by weight scale) • developer checks MongoDB version, bson library, Python version — 3 weeks • fix: return only scalar metrics • model.save(f'models/trial_{tid}.h5') separately • BSON doc 16.8 MB→2 KB • fmin() completes all 300 trials • 1.5h retainer engagement

Eastfield Hyperopt ML Ltd. ran a 300-trial TPE hyperparameter search over a Keras LSTM for time-series forecasting using Hyperopt 0.2.7 and MongoDB 6.0 as the MongoTrials backend. Their objective function trained the LSTM, computed validation loss, and then — to ensure the best model could be recovered after the study — stored the trained model weights directly in the return dict: {'loss': val_loss, 'status': STATUS_OK, 'weights': model.get_weights(), 'model_config': model.get_config()}. The first 149 trials completed correctly. At trial 150, fmin() raised bson.errors.BSONError: BSON document too large (16793616) and the study stopped. All subsequent trials were marked FAILED. The developer engaged a Hyperopt retainer developer after three weeks.

The objective function at the time of the engagement:

# Eastfield Hyperopt ML Ltd. — Hyperopt MongoTrials study
# hyperopt 0.2.7, Python 3.11, MongoDB 6.0

import hyperopt
from hyperopt import fmin, tpe, hp, STATUS_OK, Trials
from hyperopt.mongoexp import MongoTrials
import numpy as np

trials = MongoTrials(
    "mongo://localhost:27017/eastfield_hpo/jobs",
    exp_key="lstm_forecast_v4",
)

space = {
    "lstm_units": hp.quniform("lstm_units", 32, 512, 32),
    "num_layers": hp.choice("num_layers", [1, 2, 3]),
    "dropout": hp.uniform("dropout", 0.0, 0.5),
    "learning_rate": hp.loguniform("lr", np.log(1e-5), np.log(1e-2)),
    "batch_size": hp.choice("batch_size", [16, 32, 64, 128]),
    "recurrent_dropout": hp.uniform("recurrent_dropout", 0.0, 0.3),
}

def objective(params: dict) -> dict:
    model = build_lstm(
        lstm_units=int(params["lstm_units"]),
        num_layers=params["num_layers"],
        dropout=params["dropout"],
        recurrent_dropout=params["recurrent_dropout"],
    )
    optimizer = tf.keras.optimizers.Adam(learning_rate=params["learning_rate"])
    model.compile(optimizer=optimizer, loss="mse")

    history = model.fit(
        X_train, y_train,
        batch_size=int(params["batch_size"]),
        epochs=50,
        validation_data=(X_val, y_val),
        verbose=0,
    )
    val_loss = min(history.history["val_loss"])

    # ← Problem: model weights stored in return dict
    return {
        "loss": val_loss,
        "status": STATUS_OK,
        "weights": model.get_weights(),       # ← 14 float32 arrays, ~15.8 MB total
        "model_config": model.get_config(),   # ← adds ~40 KB
    }
    # bson.encode(result_dict) = 16,793,616 bytes = 16.8 MB
    # MongoDB 16 MB BSON document limit → BSONError at trial 150

best = fmin(
    fn=objective,
    space=space,
    algo=tpe.suggest,
    max_evals=300,
    trials=trials,
)

# Failure at trial 150:
# bson.errors.BSONError: BSON document too large (16793616).
#   The size must be less than or equal to 16793600.
# Traceback (most recent call last):
#   File "hyperopt/mongoexp.py", line 782, in insert_trial_docs
#     self.db.jobs.insert_many(jobs)
#   ...
# Trials 1–149: STATUS_OK (document sizes 15.6–15.9 MB per trial,
#   depending on weight initialization scale — just under 16 MB)
# Trial 150: STATUS_FAIL (document 16.8 MB — weight scale slightly larger → BSONError)

Why MongoTrials has a 16 MB document limit

MongoDB enforces a hard 16 MB limit on every BSON document. This limit applies to the entire trial document that Hyperopt writes to the jobs collection, which includes Hyperopt’s internal bookkeeping fields (tid, spec, state, owner, book_time, refresh_time, misc, exp_key — roughly 500 bytes) plus the serialized return value of the objective function stored in the result subdocument.

When an objective function returns model.get_weights(), the return value is a list of NumPy float32 arrays representing every weight matrix and bias vector in the model. For the LSTM architecture in this study (512 units × 3 layers × 4 gates per LSTM cell × float32), the total parameter count is approximately 4 million, consuming 4M × 4 bytes = 16 MB when serialized to BSON binary. Combined with Hyperopt’s overhead, the trial document exceeds the 16 MB limit.

The deceptive aspect: trials 1–149 succeed because the specific weight values at trial initialization produce slightly smaller serialized sizes. BSON encodes floating-point values without compression, but NumPy arrays may contain zeros or small values that serialize to fewer bytes. The first 149 trials happen to produce documents of 15.6–15.9 MB — just under the limit. Trial 150’s specific weight initialization scales produce a document of 16.8 MB. This randomness makes the failure look non-reproducible, leading to suspicion of MongoDB version changes, driver bugs, or transient connection issues rather than a structural document-size problem.

# Diagnostic: check result dict serialized size before MongoDB insertion

import bson
import numpy as np

# Simulate the return value that caused the BSONError:
# (Use the same LSTM architecture — 512 units, 3 layers)
model = build_lstm(lstm_units=512, num_layers=3, dropout=0.3, recurrent_dropout=0.1)
weights = model.get_weights()
config = model.get_config()

result_dict = {
    "loss": 0.0312,
    "status": "ok",
    "weights": weights,         # list of 14 NumPy arrays
    "model_config": config,
}

# Measure serialized size
serialized = bson.encode(result_dict)
size_bytes = len(serialized)
print(f"Serialized result_dict size: {size_bytes:,} bytes ({size_bytes / 1e6:.1f} MB)")
# Output: Serialized result_dict size: 16,793,616 bytes (16.8 MB)

# MongoDB limit:
MONGODB_BSON_LIMIT = 16 * 1024 * 1024  # 16,777,216 bytes = 16 MB
print(f"MongoDB BSON limit: {MONGODB_BSON_LIMIT:,} bytes ({MONGODB_BSON_LIMIT / 1e6:.1f} MB)")
print(f"Exceeds limit: {size_bytes > MONGODB_BSON_LIMIT}")
# Output: Exceeds limit: True

# Check weight array sizes individually:
total_params = 0
for i, w in enumerate(weights):
    total_params += w.size
    print(f"  Layer {i}: shape={w.shape}, dtype={w.dtype}, size={w.nbytes:,} bytes ({w.nbytes/1e6:.1f} MB)")
# Output:
#   Layer 0 (LSTM kernel):       shape=(1, 2048), dtype=float32, size=8,192 bytes (0.0 MB)
#   Layer 1 (LSTM recurrent):    shape=(512, 2048), dtype=float32, size=4,194,304 bytes (4.2 MB)
#   Layer 2 (LSTM bias):         shape=(2048,), dtype=float32, size=8,192 bytes (0.0 MB)
#   ...
#   Total weight arrays: ~15.8 MB (varies ±0.5 MB by initialization scale)

# Note: BSONError occurs when total serialized size > 16,777,216 bytes
# Trials 1-149: total weight bytes 15.6-15.9 MB → document 15.7-16.0 MB → under limit
# Trial 150:    total weight bytes 16.4 MB → document 16.8 MB → BSONError

The fix: return only scalar metrics; save model to disk separately

The fix is to remove all binary payloads from the objective function’s return dict and save model artifacts to disk (or S3) independently. The return dict should contain only scalar values: loss, status, and any lightweight metrics. The trained model is saved with model.save(f'models/trial_{trial_id}.keras') inside the objective function before returning, using the trial’s _tid from the trials object to construct a unique filename. After the study completes, the best trial’s model can be loaded with tf.keras.models.load_model(best_model_path).

# Fix: return only scalar metrics; save model artifacts to disk
# hyperopt 0.2.7, Eastfield Hyperopt ML Ltd.

import os
import bson
import hyperopt
from hyperopt import fmin, tpe, hp, STATUS_OK
from hyperopt.mongoexp import MongoTrials

os.makedirs("/tmp/eastfield_models/lstm_forecast_v4", exist_ok=True)

trials = MongoTrials(
    "mongo://localhost:27017/eastfield_hpo/jobs",
    exp_key="lstm_forecast_v4",
)

space = {
    "lstm_units": hp.quniform("lstm_units", 32, 512, 32),
    "num_layers": hp.choice("num_layers", [1, 2, 3]),
    "dropout": hp.uniform("dropout", 0.0, 0.5),
    "learning_rate": hp.loguniform("lr", np.log(1e-5), np.log(1e-2)),
    "batch_size": hp.choice("batch_size", [16, 32, 64, 128]),
    "recurrent_dropout": hp.uniform("recurrent_dropout", 0.0, 0.3),
}

def objective_fixed(params: dict) -> dict:
    model = build_lstm(
        lstm_units=int(params["lstm_units"]),
        num_layers=params["num_layers"],
        dropout=params["dropout"],
        recurrent_dropout=params["recurrent_dropout"],
    )
    optimizer = tf.keras.optimizers.Adam(learning_rate=params["learning_rate"])
    model.compile(optimizer=optimizer, loss="mse")

    history = model.fit(
        X_train, y_train,
        batch_size=int(params["batch_size"]),
        epochs=50,
        validation_data=(X_val, y_val),
        verbose=0,
    )
    val_loss = min(history.history["val_loss"])

    # Save model to disk — reference by filename in result dict
    trial_id = len(trials.trials)   # current trial index
    model_path = f"/tmp/eastfield_models/lstm_forecast_v4/trial_{trial_id:04d}.keras"
    model.save(model_path)

    # Return only scalar metrics — no binary payloads
    result = {
        "loss": val_loss,
        "status": STATUS_OK,
        "val_loss": val_loss,
        "train_loss": min(history.history["loss"]),
        "model_path": model_path,   # lightweight string, not model bytes
    }

    # Verify size before MongoDB insertion (optional safeguard):
    assert len(bson.encode(result)) < 15_000_000, "Result dict too large for MongoDB"

    return result
    # bson.encode(result) ≈ 2,100 bytes (was 16,793,616 bytes)

best = fmin(
    fn=objective_fixed,
    space=space,
    algo=tpe.suggest,
    max_evals=300,
    trials=trials,
)

# Retrieve best model path from trials:
best_trial = trials.best_trial
best_model_path = best_trial["result"]["model_path"]
best_model = tf.keras.models.load_model(best_model_path)
print(f"Best val_loss: {best_trial['result']['val_loss']:.4f}")
print(f"Best model loaded from: {best_model_path}")

# Result after fix:
# bson.encode(result_dict) = 2,100 bytes (was 16.8 MB)
# fmin() completes all 300 trials: zero BSONError
# Best trial: lstm_units=448, num_layers=2, dropout=0.18,
#             learning_rate=0.0024, batch_size=64, recurrent_dropout=0.08
# Best val_loss: 0.0312

Validation

# Validation log — November 2026
# hyperopt 0.2.7, Python 3.11, MongoDB 6.0, MongoTrials
# 300-trial TPE search over Keras LSTM (512 units, 3 layers)

# ---- Before fix: model.get_weights() in return dict ----
# bson.encode(result_dict): 16,793,616 bytes = 16.8 MB
# MongoDB BSON limit: 16,777,216 bytes = 16.0 MB
# Trials completed: 149 / 300 (trials 1-149 had weights serializing to <16 MB)
# BSONError at trial 150
# Study result: incomplete; best of 149 trials only

# ---- After fix: scalar metrics only; model.save() to disk ----
# bson.encode(result_dict): 2,100 bytes = 0.002 MB
# BSONError: 0
# Trials completed: 300 / 300
# Best trial: lstm_units=448, num_layers=2, dropout=0.18,
#             learning_rate=0.0024, batch_size=64, recurrent_dropout=0.08
# Best val_loss: 0.0312
# Model files saved: /tmp/eastfield_models/lstm_forecast_v4/trial_0000.keras
#                    ... trial_0299.keras (300 files)
# Disk used by model files: 300 × 61 MB = 18.3 GB on /tmp (448-unit 2-layer LSTM)
# Note: model files should use persistent storage (NFS, S3) not /tmp in production

# ---- Hour breakdown ----
# Triage:     0.5h — BSONError at trial 150; bson.encode(result_dict): 16.8 MB;
#             identified model.get_weights() as the large payload
# Root cause: 0.25h — model.get_weights() returns 15.8 MB of float32 arrays;
#             MongoDB rejects BSON documents larger than 16 MB
# Fix:        0.25h — removed weights and model_config from return dict;
#             added model.save() with trial_id filename before return
# Validation: 0.5h — 300-trial run; zero BSONError; best val_loss 0.0312;
#             all 300 models saved to disk; 1.5h total

Related retainer posts

Track retainer hours without a client portal

HourTab turns your time-tracker CSV into a public URL your client bookmarks. No login. No portal. CSV in, URL out.

See pricing →