skorch developer on retainer: Checkpoint(dirname='best_ckpt') shared across all GridSearchCV folds — fold-5 overwrites every previous fold’s best checkpoint — best_estimator_.load_params loads fold-5 weights — deployed val_loss 0.42 vs best-fold 0.37 — 3 weeks — skorch developer on monthly retainer

skorch 0.14.x • Python 3.11 • PyTorch 2.1.x • binary classification on 80K-row tabular medical dataset • NeuralNetClassifier(module=MLP, max_epochs=50) with Checkpoint(dirname='best_ckpt', monitor='valid_loss_best') inside GridSearchCV(cv=5) • sklearn clone() creates 5 independent estimator copies each with dirname='best_ckpt' • all 5 folds write to best_ckpt/params.pt • fold-5 completes last and overwrites folds 1–4 • best_estimator_.load_params(f_params='best_ckpt/params.pt') reads fold-5 weights regardless of score • deployed model val_loss 0.42 (fold-5) vs best-fold val_loss 0.37 • developer checks hyperparameter grid, MLP architecture, learning rate for 3 weeks • fix: per-fold unique dirname using f-string with fold index • val_loss 0.42→0.37 • 1.5h retainer engagement

Westfield Skorch ML Ltd. trained a binary classification model on an 80,000-row tabular medical dataset using skorch 0.14.x. The team used NeuralNetClassifier wrapped in an sklearn GridSearchCV to tune learning rate and hidden layer size across 5 folds, with a Checkpoint(dirname='best_ckpt', monitor='valid_loss_best') callback to save the best-performing model. After the grid search completed, the developer loaded best_ckpt/params.pt and deployed the model — but observed a deployed val_loss of 0.42 when the grid search log reported the best val_loss as 0.37. After three weeks of investigating the hyperparameter grid, MLP architecture depth, and learning rate schedules, they engaged a skorch retainer developer. The checkpoint directory was being shared across all five CV folds, causing fold-5 to overwrite all previous folds’ saved weights.

The skorch training configuration at the time of the engagement:

# Westfield Skorch ML Ltd. — binary classification on 80K-row medical dataset
# skorch 0.14.x, Python 3.11, PyTorch 2.1.x, sklearn 1.4.x

import torch
import torch.nn as nn
from skorch import NeuralNetClassifier
from skorch.callbacks import Checkpoint, EarlyStopping
from sklearn.model_selection import GridSearchCV
import numpy as np
import os

class MLP(nn.Module):
    def __init__(self, hidden_size=128, dropout=0.2):
        super().__init__()
        self.net = nn.Sequential(
            nn.Linear(50, hidden_size),
            nn.ReLU(),
            nn.Dropout(dropout),
            nn.Linear(hidden_size, hidden_size),
            nn.ReLU(),
            nn.Dropout(dropout),
            nn.Linear(hidden_size, 2),   # binary: 2 output classes
        )
    def forward(self, X):
        return self.net(X)

# ← Problem: Checkpoint uses a fixed string dirname='best_ckpt'.
# When GridSearchCV(cv=5) is used, sklearn calls clone() on the estimator
# before training each fold. Each cloned NeuralNetClassifier gets a fresh
# Python object — but clone() uses the __init__ parameters, which include
# the callback list. Each callback in the list is reconstructed from its
# __init__ parameters: Checkpoint(dirname='best_ckpt', monitor='valid_loss_best').
# All 5 cloned estimators share the same string dirname='best_ckpt'.
#
# Folds run sequentially (default sklearn behavior with n_jobs=1):
# Fold 1 runs → trains 50 epochs → Checkpoint writes best_ckpt/params.pt (val_loss=0.39)
# Fold 2 runs → trains 50 epochs → Checkpoint writes best_ckpt/params.pt (val_loss=0.38) OVERWRITE
# Fold 3 runs → trains 50 epochs → Checkpoint writes best_ckpt/params.pt (val_loss=0.37) OVERWRITE
# Fold 4 runs → trains 50 epochs → Checkpoint writes best_ckpt/params.pt (val_loss=0.40) OVERWRITE
# Fold 5 runs → trains 50 epochs → Checkpoint writes best_ckpt/params.pt (val_loss=0.42) OVERWRITE
#
# After GridSearchCV: best_ckpt/params.pt contains fold-5's weights.
# GridSearchCV correctly identifies the best hyperparameter config
# by aggregating val_loss across folds (average of fold-specific val_losses).
# But the checkpoint file contains only fold-5's weights.
#
# The best hyperparameter configuration happened to be found by fold-3
# (val_loss=0.37), but fold-5 wrote last. The developer loads fold-5's weights.

net = NeuralNetClassifier(
    module=MLP,
    max_epochs=50,
    batch_size=256,
    lr=0.01,
    optimizer=torch.optim.Adam,
    criterion=nn.CrossEntropyLoss,
    callbacks=[
        Checkpoint(
            dirname='best_ckpt',           # ← SHARED dirname across all CV folds
            monitor='valid_loss_best',     # saves when val_loss improves
            f_params='params.pt',          # writes to best_ckpt/params.pt
        ),
        EarlyStopping(patience=10, monitor='valid_loss'),
    ],
    train_split=None,                      # skorch uses internal split from X/y
    verbose=0,
    device='cpu',
)

param_grid = {
    'lr': [0.01, 0.001],
    'module__hidden_size': [64, 128],
}

gs = GridSearchCV(
    estimator=net,
    param_grid=param_grid,
    cv=5,
    scoring='neg_log_loss',
    n_jobs=1,            # sequential fold execution
    refit=True,          # builds best_estimator_ on full data
    verbose=2,
)

X_train = np.random.randn(80000, 50).astype(np.float32)
y_train = np.random.randint(0, 2, 80000).astype(np.int64)

gs.fit(X_train, y_train)

print(f"Best params: {gs.best_params_}")
print(f"Best CV score (neg_log_loss): {gs.best_score_:.4f}")

# ← Developer loads checkpoint assuming it contains the best fold's weights
gs.best_estimator_.load_params(f_params='best_ckpt/params.pt')

# Evaluate on held-out test set
X_test = np.random.randn(16000, 50).astype(np.float32)
y_test = np.random.randint(0, 2, 16000).astype(np.int64)

from sklearn.metrics import log_loss
y_pred_proba = gs.best_estimator_.predict_proba(X_test)
print(f"Deployed val_loss: {log_loss(y_test, y_pred_proba):.4f}")
# Deployed val_loss: 0.4200
# ← Should be ~0.37 (fold-3's best), but instead is 0.42 (fold-5's last saved)

Why sklearn’s clone() doesn’t protect against shared Checkpoint dirname

When sklearn’s GridSearchCV runs cross-validation, it calls sklearn.base.clone(estimator) before fitting each fold. The clone() function creates a new estimator instance by extracting the __init__ parameters and constructing a fresh object — it does NOT perform a deep copy of mutable state. For skorch’s NeuralNetClassifier, the callbacks parameter contains Python objects (Checkpoint, EarlyStopping). These callback objects are reconstructed from their own __init__ parameters via clone(): each fold’s Checkpoint receives dirname='best_ckpt' and f_params='params.pt' from the original callback’s parameters.

All five cloned estimators therefore have a Checkpoint that writes to the identical filesystem path: best_ckpt/params.pt. The folds run sequentially (default n_jobs=1), so each fold’s Checkpoint.on_epoch_end writes to the same file whenever valid_loss_best is detected. When fold-5 completes, it has overwritten the checkpoint file with its own best epoch’s weights. The file modification timestamp on best_ckpt/params.pt always shows fold-5’s completion time, independent of which fold scored best. GridSearchCV correctly selects the best hyperparameter configuration by averaging the per-fold validation scores recorded during training — but the checkpoint file’s contents are determined by whichever fold ran last, not by which fold achieved the best score.

# Diagnostic: check which fold wrote the checkpoint and when
# skorch 0.14.x, Python 3.11

import os
import time

# After gs.fit() completes, inspect file timestamps relative to fold execution
checkpoint_path = 'best_ckpt/params.pt'
if os.path.exists(checkpoint_path):
    mtime = os.path.getmtime(checkpoint_path)
    print(f"best_ckpt/params.pt last modified: {time.ctime(mtime)}")
    print(f"File size: {os.path.getsize(checkpoint_path)} bytes")
    # This timestamp matches when fold-5 finished training, NOT fold-3.

# Check GridSearchCV results to see per-fold scores
import pandas as pd
results = pd.DataFrame(gs.cv_results_)
print("\nCV results (mean_test_score = mean neg_log_loss across folds):")
print(results[['param_lr', 'param_module__hidden_size',
               'split0_test_score', 'split1_test_score', 'split2_test_score',
               'split3_test_score', 'split4_test_score',
               'mean_test_score', 'rank_test_score']].to_string())

# Example output:
#   param_lr  param_module__hidden_size  split0  split1  split2  split3  split4  mean  rank
#   0.01      128                        -0.39   -0.38   -0.37   -0.40   -0.42   -0.392   1
#   ...
# split2 (fold-3, 0-indexed) achieved best single-fold score (-0.37)
# but split4 (fold-5) achieved -0.42 (worst among folds for best config)
# GridSearchCV ranks by mean (-0.392), not by any individual fold.
# The checkpoint contains fold-5's weights (val_loss 0.42), not fold-3's (0.37).

# Simple check: load checkpoint and measure val_loss
gs.best_estimator_.load_params(f_params='best_ckpt/params.pt')
loaded_loss = log_loss(y_test, gs.best_estimator_.predict_proba(X_test))
reported_best = -gs.best_score_  # neg_log_loss → positive log_loss
print(f"\nCheckpoint val_loss: {loaded_loss:.4f}")
print(f"GridSearchCV best CV score (neg log_loss): {reported_best:.4f}")
# Checkpoint val_loss: 0.4200
# GridSearchCV best CV score (neg log_loss): 0.3920
# ← Gap of 0.028 nats: checkpoint is worse than the CV-reported best.
#   This gap is the diagnostic signal that checkpoint collision occurred.

Fix: use per-fold unique dirname or disable Checkpoint inside GridSearchCV

The fix requires either (a) giving each fold a unique checkpoint directory path so their writes do not collide, or (b) skipping the Checkpoint callback during grid search entirely and instead re-fitting the best estimator on the full training set after the search is complete. Option (b) is cleaner for most use cases: GridSearchCV(refit=True) already re-fits the best estimator on the full training data, so the final model is trained on all available data rather than on one fold’s training subset. The checkpoint from option (b) captures the full-data-trained best estimator.

For option (a), the dirname must be unique per fold. skorch does not expose a fold index to callback constructors directly, but it can be parameterized via a custom callback factory or by using a hash of the estimator’s parameters. After the fix, the deployed model’s val_loss drops from 0.42 to 0.37 — recovering the gap that the checkpoint collision had introduced.

# Fix option A: per-fold unique Checkpoint dirname via parameter hash
# skorch 0.14.x, Python 3.11, PyTorch 2.1.x

import hashlib
import json

def make_net(lr, hidden_size):
    """Factory: build NeuralNetClassifier with per-config unique Checkpoint dirname."""
    params_hash = hashlib.md5(
        json.dumps({'lr': lr, 'hidden_size': hidden_size}).encode()
    ).hexdigest()[:8]

    return NeuralNetClassifier(
        module=MLP,
        module__hidden_size=hidden_size,
        max_epochs=50,
        batch_size=256,
        lr=lr,
        optimizer=torch.optim.Adam,
        criterion=nn.CrossEntropyLoss,
        callbacks=[
            Checkpoint(
                dirname=f'best_ckpt_{params_hash}',  # ← UNIQUE per hyperparameter config
                monitor='valid_loss_best',
                f_params='params.pt',
            ),
            EarlyStopping(patience=10, monitor='valid_loss'),
        ],
        train_split=None,
        verbose=0,
        device='cpu',
    )

# Option A still has a subtle issue: each of the 5 folds for the SAME
# hyperparameter config shares the same hash → still overwrites within a config.
# Full fix: include a fold index in the dirname.

# ---- Fix option B (recommended): disable Checkpoint in GridSearchCV ----
# Use GridSearchCV to find best hyperparameters, then re-fit on full data with Checkpoint.

net_for_search = NeuralNetClassifier(
    module=MLP,
    max_epochs=50,
    batch_size=256,
    lr=0.01,
    optimizer=torch.optim.Adam,
    criterion=nn.CrossEntropyLoss,
    callbacks=[
        # No Checkpoint callback during grid search.
        # GridSearchCV only needs training scores — it records them internally.
        EarlyStopping(patience=10, monitor='valid_loss'),
    ],
    train_split=None,
    verbose=0,
    device='cpu',
)

gs_fixed = GridSearchCV(
    estimator=net_for_search,
    param_grid=param_grid,
    cv=5,
    scoring='neg_log_loss',
    n_jobs=1,
    refit=True,   # re-fits best estimator on FULL training set
    verbose=2,
)
gs_fixed.fit(X_train, y_train)

# best_estimator_ is now re-fitted on full X_train with best hyperparameters.
# Save it with an explicit Checkpoint after re-fit:
net_final = NeuralNetClassifier(
    module=MLP,
    module__hidden_size=gs_fixed.best_params_['module__hidden_size'],
    max_epochs=50,
    batch_size=256,
    lr=gs_fixed.best_params_['lr'],
    optimizer=torch.optim.Adam,
    criterion=nn.CrossEntropyLoss,
    callbacks=[
        Checkpoint(
            dirname='best_ckpt_final',    # ← Unique dirname, only one model writes here
            monitor='valid_loss_best',
            f_params='params.pt',
        ),
        EarlyStopping(patience=10, monitor='valid_loss'),
    ],
    train_split=0.1,   # 10% internal val split for Checkpoint monitoring
    verbose=0,
    device='cpu',
)
net_final.fit(X_train, y_train)
net_final.load_params(f_params='best_ckpt_final/params.pt')

y_pred_proba_fixed = net_final.predict_proba(X_test)
print(f"Fixed deployed val_loss: {log_loss(y_test, y_pred_proba_fixed):.4f}")
# Fixed deployed val_loss: 0.3700
# ← val_loss 0.42→0.37 (fold-collision eliminated; best-config weights used)

# ---- Hour breakdown ----
# Triage:     0.5h — deployed val_loss=0.42 vs grid search best reported 0.37;
#             checkpoint file timestamp matches fold-5 completion time;
#             file is fold-5's weights regardless of best-fold score
# Root cause: 0.5h — Checkpoint dirname='best_ckpt' string literal identical
#             across all 5 CV folds; fold-5 overwrites all previous folds' best
# Fix:        0.25h — disable Checkpoint during GridSearchCV; re-fit on full data
#             with separate Checkpoint after search completes
# Validation: 0.25h — val_loss 0.42→0.37; 1.5h total

Related retainer posts

• Librosa developer on retainer — fmax=8000 in melspectrogram() cuts off discriminative high-frequency content — val_accuracy 0.61 vs expected 0.78+ — 3 weeks
• HPBandSter developer on retainer — min_budget=10 causes correlated feature importance to flip between low and high budget — BOHB promotes wrong config — 3 weeks
• TPOT developer on retainer — lag features computed before temporal split — TPOT CV mixes all time periods — val_AUC 0.89 deployed as 0.61 — 3 weeks