MLflow developer on retainer: mlflow.sklearn.autolog() drops 17 of 20 GridSearchCV hyperparameter runs — SQLite database is locked under n_jobs=-1 — production model deployed with suboptimal hyperparameters — MLflow developer on monthly retainer

MLflow 2.12.2 • AWS EC2 c5.2xlarge (8 vCPUs, 16 GB RAM) • Python 3.11 • scikit-learn 1.4.2 • mlflow.sklearn.autolog() • GridSearchCV(n_jobs=-1) • SQLite at /mnt/efs/mlruns/mlruns.db (NFS-mounted EFS) • sqlite3.OperationalError: database is locked • 17 of 20 runs silently dropped • retainer billing

On November 12, 2026, a new data scientist at Meridian Health Analytics Corp. re-ran the team’s patient readmission model training script as a routine familiarization exercise. She had a habit of setting n_jobs=1 when exploring unfamiliar code — parallel jobs made log output harder to read. With n_jobs=1, the training script produced 20 MLflow runs, all with populated hyperparameters and metrics. When she compared the best run from this execution to the model currently in production, the numbers did not match: the production model had n_estimators=50, max_depth=3, and a cross-validated AUC of 0.803. Her run showed a best model of n_estimators=200, max_depth=7, and an AUC of 0.847. The production model had been deployed on July 14, 2026 — four months earlier. An MLflow developer retainer was engaged to investigate why the original training run had produced only 3 complete runs instead of 20, and whether the production model had been selected from an incomplete grid search.

Meridian Health Analytics ran their ML training workloads on AWS EC2 c5.2xlarge instances (8 vCPUs, 16 GB RAM). The MLflow tracking server used a SQLite backend at /mnt/efs/mlruns/mlruns.db — a file hosted on an NFS-mounted EFS volume shared across the ML team so that multiple data scientists could access the same experiment tracking database from their respective training instances. The training script, train_patient_readmission_model.py, called mlflow.sklearn.autolog() at the top of the file and then ran a GridSearchCV with GradientBoostingClassifier across 20 hyperparameter combinations (4 n_estimators values × 5 max_depth values) with cv=5 and n_jobs=-1.

# train_patient_readmission_model.py — MLflow 2.12.2, Python 3.11, scikit-learn 1.4.2
# As run on July 14, 2026 to select the production model

import mlflow
import mlflow.sklearn
import pandas as pd
from sklearn.ensemble import GradientBoostingClassifier
from sklearn.model_selection import GridSearchCV, train_test_split
from sklearn.metrics import roc_auc_score

# MLflow tracking: SQLite backend on NFS-mounted EFS volume
# Shared across the ML team for collaborative experiment tracking
MLFLOW_TRACKING_URI = "sqlite:////mnt/efs/mlruns/mlruns.db"
mlflow.set_tracking_uri(MLFLOW_TRACKING_URI)
mlflow.set_experiment("patient_readmission_v3")

# mlflow.sklearn.autolog() patches sklearn's fit() to automatically log:
# - hyperparameters (estimator params)
# - metrics (training score, cross-validation scores)
# - the fitted model as an artifact
# Wraps all logging calls in try: ... except Exception: logger.warning(...)
mlflow.sklearn.autolog()  # ← AUTOLOG ENABLED — BEFORE GridSearchCV

df = pd.read_parquet("s3://meridian-ml-data/patient_readmission_features.parquet")
X = df.drop(columns=["readmitted_30d"])
y = df["readmitted_30d"]
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)

param_grid = {
    "n_estimators": [50, 100, 150, 200],   # 4 values
    "max_depth":    [3, 5, 7, 9, 11],       # 5 values
}
# 4 × 5 = 20 hyperparameter combinations
# cv=5: 5-fold cross-validation per combination
# n_jobs=-1: use all available CPUs — on c5.2xlarge, spawns 8 joblib worker processes
# Each worker process independently calls the autolog-patched fit()
# Each autolog fit() call attempts to write to SQLite at /mnt/efs/mlruns/mlruns.db
# SQLite file-level locking: only ONE writer at a time
# 8 concurrent writers → 7 hit sqlite3.OperationalError: database is locked (5s timeout)
# MLflow autolog catches the exception silently → run created, no params/metrics logged
grid_search = GridSearchCV(
    estimator=GradientBoostingClassifier(random_state=42),
    param_grid=param_grid,
    cv=5,
    scoring="roc_auc",
    n_jobs=-1,  # ← THE BUG: spawns 8 joblib workers on 8-vCPU c5.2xlarge
    verbose=1,
)

grid_search.fit(X_train, y_train)

# This line runs fine — GridSearchCV trained all 20 combinations correctly
# The models were trained. Only the MLflow logging was silently dropped.
best_model = grid_search.best_estimator_
test_auc = roc_auc_score(y_test, best_model.predict_proba(X_test)[:, 1])
print(f"Best params: {grid_search.best_params_}")
print(f"Best CV AUC: {grid_search.best_score_:.4f}")
print(f"Test AUC:    {test_auc:.4f}")

# Output on July 14, 2026:
# Best params: {'max_depth': 3, 'n_estimators': 50}   ← WRONG: only 3 of 20 runs logged
# Best CV AUC: 0.8031                                  ← WRONG: actual best is 0.847
# Test AUC:    0.7918

The n_jobs=-1 parameter in GridSearchCV instructs joblib to use all available CPUs. On the c5.2xlarge instance with 8 vCPUs, joblib spawned 8 worker processes. Each worker process trained a different subset of the hyperparameter grid with 5-fold cross-validation. At the end of each fold and each candidate fit, MLflow’s autolog-patched fit() method attempted to write the hyperparameters and metrics back to the SQLite tracking database at /mnt/efs/mlruns/mlruns.db. Because this file lived on an NFS-mounted EFS volume, the SQLite file-locking mechanism operated over NFS — a configuration that SQLite itself documents as problematic for concurrent access, but one that is common in ML team shared-storage setups.

SQLite’s write behavior is serialized by a file-level lock. When one process holds the write lock, every other process attempting to write will block and retry until either the lock is released or the busy timeout expires. MLflow’s SQLAlchemy connection configuration uses a 5-second busy timeout. With 8 worker processes attempting simultaneous writes, 7 processes typically exhausted the 5-second timeout before the first writer released the lock. Each timeout produced a sqlite3.OperationalError: database is locked exception inside the autolog logging code. MLflow’s autolog infrastructure wraps all logging calls in a try: ... except Exception: logger.warning(...) block, which is by design — autolog is intended to be non-fatal so that a logging failure does not crash training. The practical effect in this scenario was that the exception was swallowed, a WARNING was emitted to the Python logger (which was not monitored), and the hyperparameter combination’s MLflow run was created in the database with a run_id and status RUNNING but no logged params, metrics, or tags.

Why the MLflow UI showed 3 complete runs instead of 0 or 20

The failure mode produced a deceptive result in the MLflow UI. Because MLflow’s autolog creates a run at the start of each candidate fit and then logs params and metrics at the end, 20 runs were created in the database — one per hyperparameter combination. Of those 20 runs, the 3 worker processes that happened to acquire the SQLite write lock sequentially (rather than concurrently) successfully logged their params and metrics. The other 17 runs existed in the database with valid run_ids and a FINISHED status but contained no logged data.

In the MLflow UI, the experiment view showed all 20 runs. When the ML team sorted by metrics.training_roc_auc to find the best model, the 17 empty runs sorted to the bottom (no metric value) and the 3 complete runs sorted to the top. The team selected n_estimators=50, max_depth=3, CV AUC 0.803 as their best model — the best of the 3 runs that happened to log. The team had no reason to question completeness: they saw 20 runs, the UI sorted by metric, and the top result looked reasonable for a readmission prediction task. The model was registered in the MLflow Model Registry and deployed to production on July 14, 2026.

Diagnosing the silent drop: querying MLflow’s SQLite database directly

The investigation began in the MLflow database itself. Rather than using the MLflow UI or Python API, the retainer developer queried the SQLite database directly to examine run completeness across all 20 candidates. The query joined the runs table against the params and metrics tables to show which runs had logged data and which had been created but remained empty.

-- Diagnostic: identify MLflow runs with missing params/metrics
-- Run against /mnt/efs/mlruns/mlruns.db after November 12, 2026 discovery
-- Targets experiment 'patient_readmission_v3', run_id from the July 14 training

-- Step 1: find the experiment_id
SELECT experiment_id, name, artifact_location
FROM experiments
WHERE name = 'patient_readmission_v3';
-- Returns: experiment_id = 3

-- Step 2: count params and metrics per run
SELECT
    r.run_uuid,
    r.status,
    r.start_time,
    COUNT(DISTINCT p.key)  AS param_count,
    COUNT(DISTINCT m.key)  AS metric_count
FROM runs r
LEFT JOIN params  p ON r.run_uuid = p.run_uuid
LEFT JOIN metrics m ON r.run_uuid = m.run_uuid
WHERE r.experiment_id = '3'
  AND r.start_time BETWEEN 1752451200000 AND 1752537600000  -- July 14, 2026 UTC ms
GROUP BY r.run_uuid, r.status, r.start_time
ORDER BY param_count DESC, r.start_time;

-- Results (truncated to show pattern):
-- run_uuid                             status    param_count  metric_count
-- ----                                 --------  -----------  ------------
-- 3a7f2d1e8b4c9f0a...                  FINISHED  4            6           ← logged OK
-- 9c1b5e3d7a8f2c4b...                  FINISHED  4            6           ← logged OK
-- 2e8d4f6a1b3c9e7d...                  FINISHED  4            6           ← logged OK
-- 5b9c2e1f4d7a8b3c...                  FINISHED  0            0           ← EMPTY RUN
-- 7a3d8f2b1c5e9d4a...                  FINISHED  0            0           ← EMPTY RUN
-- 1f4b6d9e3c7a2b8d...                  FINISHED  0            0           ← EMPTY RUN
-- ... (14 more EMPTY runs)
--
-- Summary: 3 runs with param_count=4 / metric_count=6
--          17 runs with param_count=0 / metric_count=0

-- Step 3: check Python log output from July 14 training for the autolog warning
-- grep "autologging" /var/log/meridian-ml/train_20260714.log
-- Output (repeated 17 times):
-- WARNING mlflow.utils.autologging_utils: Encountered unexpected error during
-- sklearn autologging: (sqlite3.OperationalError) database is locked
-- [SQL: INSERT INTO params (run_uuid, key, value) VALUES (?, ?, ?)]
-- (Background on this error at: https://sqlalche.me/e/14/e3q8)

-- Step 4: confirm SQLite WAL log indicates concurrent write contention
-- ls -la /mnt/efs/mlruns/mlruns.db*
-- -rw-r--r-- 1 mlflow mlflow  14680064 Nov 12 09:14 mlruns.db
-- -rw-r--r-- 1 mlflow mlflow         0 Nov 12 09:14 mlruns.db-shm
-- -rw-r--r-- 1 mlflow mlflow         0 Nov 12 09:14 mlruns.db-wal
-- (WAL file is empty now; during the July 14 run it grew to 48 MB as
--  concurrent writers queued up and timed out)

The diagnostic confirmed the pattern immediately. Of the 20 runs created during the July 14 training, 17 had zero logged params and zero logged metrics. The Python training log from July 14 contained 17 instances of the MLflow autolog warning about sqlite3.OperationalError: database is locked. The SQLite WAL log had grown to 48 MB during the run as 8 concurrent writer processes queued their transactions — 7 of the 8 processes hit the 5-second busy timeout before the first writer released the lock. The pattern was consistent: whichever 3 worker processes happened to serialize their writes in the first 5 seconds of the GridSearchCV execution got their params logged; the remaining 17 did not.

The fix: switch to PostgreSQL backend, or use n_jobs=1 with SQLite

The fix had two options. The immediate workaround was to set n_jobs=1 in GridSearchCV, which eliminates joblib parallelism and causes all 20 candidate fits to execute sequentially in the main process. With a single writer process, SQLite file-level locking is never contested, and all 20 runs log successfully. This approach costs training time — sequential execution of 20 × 5-fold cross-validation on a c5.2xlarge took approximately 12 minutes instead of 2 minutes with n_jobs=-1 — but is the safest interim fix when a PostgreSQL backend is not yet available. The production fix was to migrate the MLflow tracking backend from SQLite to PostgreSQL, which uses row-level locking and supports concurrent writes from multiple processes without contention.

# train_patient_readmission_model.py — fixed version
# Option A: PostgreSQL backend (recommended for multi-process and multi-user workloads)
# Option B: n_jobs=1 interim fix (safe with SQLite, slower training)

import mlflow
import mlflow.sklearn
import pandas as pd
from sklearn.ensemble import GradientBoostingClassifier
from sklearn.model_selection import GridSearchCV, train_test_split
from sklearn.metrics import roc_auc_score
import os

# FIX (Option A): Switch to PostgreSQL backend
# PostgreSQL supports concurrent writes via row-level locking —
# 8 simultaneous joblib workers can each open a connection and write
# to separate rows without blocking each other.
#
# Setup steps (run once):
#   pip install psycopg2-binary
#   mlflow db upgrade postgresql://mlflow_user:password@rds-endpoint.us-east-1.rds.amazonaws.com/mlflow
#   export MLFLOW_TRACKING_URI="postgresql://mlflow_user:password@rds-endpoint.us-east-1.rds.amazonaws.com/mlflow"
#
# FIX (Option B): Keep SQLite, but force n_jobs=1
# MLFLOW_TRACKING_URI = "sqlite:////mnt/efs/mlruns/mlruns.db"
# Use n_jobs=1 in GridSearchCV (see below) — slower but correct.

MLFLOW_TRACKING_URI = os.environ.get(
    "MLFLOW_TRACKING_URI",
    "postgresql://mlflow_user:password@rds-endpoint.us-east-1.rds.amazonaws.com/mlflow",
)
mlflow.set_tracking_uri(MLFLOW_TRACKING_URI)
mlflow.set_experiment("patient_readmission_v3_fixed")

mlflow.sklearn.autolog()

df = pd.read_parquet("s3://meridian-ml-data/patient_readmission_features.parquet")
X = df.drop(columns=["readmitted_30d"])
y = df["readmitted_30d"]
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)

param_grid = {
    "n_estimators": [50, 100, 150, 200],
    "max_depth":    [3, 5, 7, 9, 11],
}

grid_search = GridSearchCV(
    estimator=GradientBoostingClassifier(random_state=42),
    param_grid=param_grid,
    cv=5,
    scoring="roc_auc",
    # FIX (Option A — PostgreSQL): n_jobs=-1 is safe; PostgreSQL handles concurrent writes
    n_jobs=-1,
    # FIX (Option B — SQLite interim): set n_jobs=1 to serialize writes to SQLite
    # n_jobs=1,
    verbose=1,
)

grid_search.fit(X_train, y_train)

# FIX: Post-training run completeness validation
# Query MLflow to confirm all 20 candidates logged params and metrics
import mlflow
from mlflow.tracking import MlflowClient

client = MlflowClient()
experiment = client.get_experiment_by_name("patient_readmission_v3_fixed")
runs = client.search_runs(
    experiment_ids=[experiment.experiment_id],
    order_by=["metrics.training_roc_auc DESC"],
)

complete_runs = [r for r in runs if r.data.params]
empty_runs    = [r for r in runs if not r.data.params]

print(f"Total runs created:    {len(runs)}")
print(f"Runs with params:      {len(complete_runs)}")
print(f"Runs without params:   {len(empty_runs)}")

# FIX: Assert all 20 combinations were logged before proceeding
expected_combinations = len(param_grid["n_estimators"]) * len(param_grid["max_depth"])
if len(complete_runs) != expected_combinations:
    raise RuntimeError(
        f"Expected {expected_combinations} complete MLflow runs, "
        f"got {len(complete_runs)} with params and {len(empty_runs)} empty. "
        f"Check MLflow backend concurrency compatibility with n_jobs setting. "
        f"If using SQLite backend, set n_jobs=1 in GridSearchCV."
    )

best_model = grid_search.best_estimator_
test_auc = roc_auc_score(y_test, best_model.predict_proba(X_test)[:, 1])
print(f"Best params: {grid_search.best_params_}")
print(f"Best CV AUC: {grid_search.best_score_:.4f}")
print(f"Test AUC:    {test_auc:.4f}")

# Output after fix (all 20 combinations logged, PostgreSQL backend):
# Total runs created:  20
# Runs with params:    20
# Runs without params: 0
# Best params: {'max_depth': 7, 'n_estimators': 200}  ← CORRECT best from full grid
# Best CV AUC: 0.8470                                  ← AUC uplift from 0.803
# Test AUC:    0.8312

With the PostgreSQL backend configured and all 20 combinations re-run, the best model was n_estimators=200, max_depth=7, CV AUC 0.847 — a 4.4-percentage-point improvement over the 0.803 AUC of the production model. For a patient readmission prediction task, an AUC improvement of this magnitude can translate directly to better clinical decisions: higher true positive rate at a fixed false positive rate means more at-risk patients flagged for intervention before discharge. The retrained model was validated on the holdout set (AUC 0.831 vs. the old model’s holdout AUC of 0.792) and redeployed on November 14, 2026.

Why MLflow autolog’s exception suppression is a silent failure pattern

The autolog exception-suppression design is documented in MLflow’s source code and is intentional. The mlflow.utils.autologging_utils module wraps all autolog callbacks in a try: ... except Exception block to prevent logging failures from crashing training. This is a reasonable design choice for individual logging calls during a long training run — you would not want a transient network error writing to a remote tracking server to abort a 6-hour GPU training job. But the same exception-suppression behavior makes the SQLite concurrency failure mode completely invisible at the script level. Training completes. The script prints the best params from grid_search.best_params_ — which reflects the best of the 20 scikit-learn GridSearchCV candidates (all 20 were trained correctly; the issue was only MLflow logging) — and exits with code 0.

The deceptive element is that grid_search.best_params_ returns the correct answer from scikit-learn’s perspective: it found the best parameters across all 20 candidates based on cross-validation scores computed entirely within the scikit-learn machinery, which does not touch MLflow. The problem is not that the wrong model was selected by GridSearchCV — GridSearchCV saw all 20 candidates and correctly identified n_estimators=200, max_depth=7 as the best. The problem is that the team did not use grid_search.best_params_ to select the production model. They used the MLflow UI, sorted by metric, and selected the best of the 3 complete runs. If the script had included a line printing grid_search.best_params_ and the team had read it, they would have seen the correct answer. Instead, the team relied on the MLflow experiment tracking view as their authoritative record of which model was best — a reasonable expectation when MLflow autolog is active — and the incomplete logging produced the wrong answer.

The 4.5-hour engagement covered investigation (1.5h — 0.5h MLflow UI analysis: 20 runs listed, only 3 with params/metrics populated; 0.25h direct SQLite query: SELECT run_uuid, status, COUNT(params), COUNT(metrics) confirms 17 runs with 0 params and 0 metrics; 0.25h Python training log review: 17 instances of WARNING mlflow.utils.autologging_utils: Encountered unexpected error during sklearn autologging: sqlite3.OperationalError: database is locked; 0.25h autolog source code review confirming try/except Exception exception-suppression in mlflow.utils.autologging_utils; 0.25h confirming all 20 GridSearchCV candidates trained correctly via grid_search.cv_results_ — bug is logging only, not training), fix (1.5h — 0.5h PostgreSQL RDS instance provisioned and mlflow db upgrade run; 0.25h MLFLOW_TRACKING_URI updated in training script and environment; 0.25h n_jobs=1 confirmed as interim fix with timing comparison; 0.5h full 20-combination re-run: all 20 runs confirmed logged; post-training completeness assertion added), and validation and re-deployment (1.5h — 0.5h best model n_estimators=200/max_depth=7 retrained on full training set; 0.5h holdout set validation: AUC 0.831 vs old model 0.792; 0.5h redeployment to production inference API and smoke test confirming new model scoring).

In a HourTab retainer log, the incident session reads: “train_patient_readmission_model.py MLflow autolog silent run-drop investigation: MLflow 2.12.2 on AWS EC2 c5.2xlarge (8 vCPUs); SQLite backend at /mnt/efs/mlruns/mlruns.db on NFS-mounted EFS; mlflow.sklearn.autolog() + GridSearchCV(GradientBoostingClassifier, 20 combinations, n_jobs=-1); 8 joblib workers hit sqlite3.OperationalError database is locked; autolog catches exception silently; 17 of 20 runs created but empty (0 params, 0 metrics); team picks best of 3 complete runs: n_estimators=50/max_depth=3/AUC 0.803; deployed July 14, 2026; actual best: n_estimators=200/max_depth=7/AUC 0.847; fix: PostgreSQL backend + run completeness assertion + re-run all 20 + redeploy; AUC uplift 0.803→0.847 confirmed on holdout; 4.5h.”

HourTab turns a time-tracker CSV into a public retainer-hours URL your client can bookmark — no login, no portal, no “how many hours do I have left?” emails. Free for one retainer. $9/mo for up to ten.

FAQ: MLflow developer retainers

See also: Apache Airflow developer retainer · Databricks developer retainer · Ray developer retainer