Neptune developer on retainer: 50-iteration hyperparameter loop crashes OOM without run.stop() — 12 Neptune runs stuck in Running state with no final AUC — Neptune developer on monthly retainer

neptune-client 1.6.2 • Python 3.11 • JupyterLab 4.1.0 • A100 GPU cluster • neptune.init_run() bare pattern • OOM at iteration 39 of 50 • run.stop() never called for 12 runs • final metadata batch not flushed • 12 runs perpetually Running • fix: context manager with neptune.init_run() as run + local CSV checkpoint • 2.5h retainer engagement

On the morning of October 21, 2026, the data science team at Haverford ML Research Ltd. opened their Neptune dashboard to review the results of a 50-iteration RandomForest hyperparameter sweep they had run overnight. The sweep had appeared to complete — no error was visible in the Jupyter notebook interface, and the cell had finished executing. But the Neptune comparison view showed only 38 of the 50 expected runs with AUC and F1 metrics. The remaining 12 runs had status Running and no metric values at all: no training loss, no validation AUC, no final F1. It looked as though those 12 iterations had started but never trained. The Neptune retainer developer was contacted at 09:15 UTC and closed the incident by 11:45 UTC.

Haverford ML Research uses Neptune (neptune-client 1.6.2) for all experiment tracking on a shared on-premises GPU cluster running JupyterLab 4.1.0. The team runs hyperparameter sweeps in Jupyter notebooks, iterating over a grid of RandomForestClassifier parameters (n_estimators, max_features, min_samples_leaf) and logging per-fold AUC, final F1, and the trained model artifact to Neptune for each iteration. The sweep in question targeted 50 parameter combinations. The notebook looped over the grid, initializing a Neptune run at the start of each iteration:

# notebooks/hyperparameter_sweep_oct2026.ipynb — original pattern (pre-incident)
# Haverford ML Research Ltd., neptune-client 1.6.2, Python 3.11

import neptune
import numpy as np
from sklearn.ensemble import RandomForestClassifier
from sklearn.model_selection import StratifiedKFold

param_grid = [
    {"n_estimators": n, "max_features": f, "min_samples_leaf": m}
    for n in [100, 200, 400, 800]
    for f in [0.3, 0.5, 0.7, 0.8, 1.0]
    for m in [1, 2, 5]
]  # 60 combinations; sweep uses first 50

trained_models = []  # accumulates all fitted models in memory — root cause

for i, params in enumerate(param_grid[:50]):
    run = neptune.init_run(
        project="haverford/classifier-sweep",
        tags=["sweep-oct-2026", f"iter-{i}"],
    )

    run["parameters"] = params

    clf = RandomForestClassifier(**params, random_state=42)
    skf = StratifiedKFold(n_splits=5, shuffle=True, random_state=42)
    aucs = []

    for fold, (train_idx, val_idx) in enumerate(skf.split(X_train, y_train)):
        clf.fit(X_train[train_idx], y_train[train_idx])
        auc = roc_auc_score(y_train[val_idx], clf.predict_proba(X_train[val_idx])[:, 1])
        run[f"train/fold_{fold}_auc"].log(auc)
        aucs.append(auc)

    final_auc = np.mean(aucs)
    final_f1 = f1_score(y_train, clf.predict(X_train), average="weighted")

    run["eval/final_auc"] = final_auc
    run["eval/final_f1"] = final_f1
    run["artifacts/model"].upload(save_model_artifact(clf, i))

    trained_models.append(clf)  # clf never deleted; memory grows each iteration

    run.stop()  # NEVER REACHED for iterations 38-49 after OOM at iteration 38

The loop ran successfully for 38 iterations, each completing in 15 to 28 minutes and calling run.stop() at the end. At iteration 38 (parameter set: n_estimators=800, max_features=0.8, min_samples_leaf=1), the cumulative memory footprint of the 38 fitted RandomForest models accumulated in the trained_models list pushed the Python process resident set size to 241 GB on the 256 GB cluster node. The Linux OOM killer sent SIGKILL to the Jupyter kernel process at 23:47 UTC. The Jupyter notebook interface displayed the cell as “completed” because the kernel restart happened at the OS level, and the browser-side UI had already received a completion signal for the outer cell before the OOM event propagated. The run.stop() call for iteration 38 was in the process of executing when SIGKILL arrived — it was not completed. Iterations 39 through 49 had been initialized by Neptune (their run entries existed in the Neptune dashboard with Created status) but had not yet started training. None of their run.stop() calls were ever reached.

The core mechanics of the data loss are straightforward. When a data scientist calls neptune.init_run(), Neptune creates a run record on its servers immediately and returns a run object backed by a local metadata buffer. Scalar metrics logged via run["eval/final_auc"] = value are written to this local buffer first and then transmitted to Neptune’s servers in batches as the run progresses. The final batch — containing the last set of logged values before run.stop() is called — is flushed to the server when run.stop() executes. Without run.stop(), the final batch remains in the local buffer and is discarded when the process terminates. For iteration 38, the final AUC and F1 values had been assigned to the run object but were in the unsynchronized buffer when SIGKILL arrived. They were never transmitted. Neptune showed the run as Running indefinitely because the server had no stop signal and no final-metrics upload.

The retainer developer confirmed the root cause in 0.5 hours by inspecting the Neptune project via the neptune API: project.fetch_runs_table() returned 50 runs, of which 38 had sys/state equal to Idle (Neptune’s post-stop state) and 12 had sys/state equal to Running. Querying run["sys/creation_time"].fetch() for each Running run showed that all 12 had creation times clustered between 23:42 and 23:58 UTC — consistent with the OOM event at 23:47 UTC. The notebook execution log confirmed that the kernel had restarted at 23:47:14 UTC. The secondary issue — 11 runs initialized but never trained (iterations 39 to 49) — was also visible in the Neptune data: those runs had no metric namespaces at all, only the parameters field that had been logged before the training loop began.

The fix had two components. First, the notebook was migrated to the context manager pattern for all Neptune run initialization. The context manager form calls run.stop() via __exit__ even when an exception propagates out of the with block, ensuring that Neptune flushes the final metadata batch before the exception continues unwinding:

# notebooks/hyperparameter_sweep_oct2026.ipynb — fixed pattern
# Haverford ML Research Ltd., neptune-client 1.6.2, Python 3.11
import gc
import csv
import neptune

CHECKPOINT_FILE = "sweep_checkpoint.csv"

for i, params in enumerate(param_grid[:50]):
    with neptune.init_run(
        project="haverford/classifier-sweep",
        tags=["sweep-oct-2026", f"iter-{i}"],
    ) as run:
        # Log run ID to local CSV before training starts.
        # If training crashes, this record lets us identify which iteration
        # was in progress and match it to the Neptune run entry.
        run_id = run["sys/id"].fetch()
        with open(CHECKPOINT_FILE, "a", newline="") as f:
            csv.writer(f).writerow([i, run_id, str(params)])

        run["parameters"] = params

        clf = RandomForestClassifier(**params, random_state=42)
        aucs = []
        for fold, (train_idx, val_idx) in enumerate(skf.split(X_train, y_train)):
            clf.fit(X_train[train_idx], y_train[train_idx])
            auc = roc_auc_score(y_train[val_idx], clf.predict_proba(X_train[val_idx])[:, 1])
            run[f"train/fold_{fold}_auc"].log(auc)
            aucs.append(auc)

        run["eval/final_auc"] = float(np.mean(aucs))
        run["eval/final_f1"] = float(f1_score(y_train, clf.predict(X_train), average="weighted"))
        run["artifacts/model"].upload(save_model_artifact(clf, i))

        # run.stop() called automatically by context manager __exit__,
        # even if an exception is raised inside the with block.

    del clf  # release model memory immediately after each iteration
    gc.collect()  # force GC to reclaim memory before next iteration

Second, the 12 orphaned Running runs were resolved via the Neptune API. The 11 runs that had never started training (iterations 39 to 49, with no metric namespaces) were deleted from the project. The run for iteration 38 — which had logged per-fold AUC values for 4 of 5 folds before the OOM event — was archived rather than deleted, since the partial fold data was useful for understanding that particular parameter set’s behavior. The sweep_checkpoint.csv file from the failed run confirmed that iterations 0 through 37 had completed and their Neptune run IDs were recoverable; the retainer developer verified that all 38 completed runs had consistent AUC values before closing the incident.

The memory growth issue — the accumulating trained_models list — was addressed by adding del clf and gc.collect() after each iteration. The original notebook had stored all 50 trained RandomForest models in memory for a planned post-sweep feature importance comparison. The retainer developer restructured this to save feature importances to Neptune at the end of each iteration (run["eval/feature_importances"] = clf.feature_importances_.tolist()) and then release the model object, reducing per-iteration memory overhead from approximately 6 GB to under 200 MB. The fixed notebook ran all 50 iterations without incident, completing in 14 hours 22 minutes with all 50 runs showing Idle status and complete AUC and F1 metrics.

Total retainer engagement: 2.5 hours (0.5h root cause: Neptune API run table query, state inspection for 12 Running runs, notebook execution log correlation with OOM timestamp; 0.5h impact assessment: which iterations were in-progress vs never started, whether partial fold-AUC data was recoverable from the partially-synced run; 1.0h fix: migrate to context manager form, add local CSV checkpoint, add del clf + gc.collect() per iteration, restructure feature importance storage to Neptune instead of in-memory list; 0.5h post-incident: archive iteration 38’s partial run, delete 11 never-trained run entries via Neptune API, document context manager pattern requirement in team wiki). Retainer billing was logged to HourTab with the neptune-client version, the OOM mechanism, and the complete fix applied, so the client could audit the full 2.5-hour work log at the shared URL without logging in.

Related retainer posts

• MLflow developer on retainer — experiment tracking and model registry retainer work
• W&B developer on retainer — Weights & Biases experiment tracking retainer work
• ClearML developer on retainer — persistent Jupyter kernel cross-task logging retainer work