ClearML developer on retainer: second Task.init() in persistent Jupyter kernel logs experiment B metrics to experiment A task — PyTorch auto-patching applies process-wide to first task — experiment B shows flat loss curves — AUC comparison inverted — wrong hyperparameters deployed to production — ClearML developer on monthly retainer

clearml 1.14.2 • PyTorch 2.1.0 • JupyterHub on GKE 1.27 • Task.init(‘churn_exp_046’) then Task.init(‘churn_exp_047’) in same kernel • PyTorch auto-patch not reset • experiment 047 metrics logged to task 046 • experiment 047 shows flat curves • AUC 0.831 selected over 0.902 • wrong hyperparameters in production • 3.5h retainer engagement

In early October 2026, the ML team at Brightfield Data Sciences completed a hyperparameter search for their quarterly churn prediction model update. The team ran 48 experiments in Jupyter notebooks over two weeks and selected the configuration with the highest validation AUC for production deployment. Three weeks after the model went live, a ClearML retainer developer conducting a routine monthly metrics audit noticed an anomaly: experiment churn_exp_046 showed a sharp loss spike at epoch 51 followed by a second convergence phase, and experiment churn_exp_047 showed completely flat loss curves with no metrics logged after initialization. The retainer developer reconstructed what had happened: the data scientist had run both experiments in the same Jupyter kernel session without restarting between runs. ClearML’s PyTorch auto-patching had applied to the entire process during the first Task.init() call and continued logging to experiment 046’s task throughout experiment 047’s training loop. Experiment 047’s correct AUC was 0.902 — better than experiment 046’s 0.887 — but the team had selected 046 because 047 appeared to have flat (failed) curves. The production model had been running with inferior hyperparameters for three weeks.

ClearML’s automatic framework detection is one of its most powerful features: it patches popular ML frameworks at the process level to capture training metrics, model checkpoints, and configuration without requiring explicit logging calls. When Task.init() is called in a Python process that uses PyTorch, ClearML installs hooks on PyTorch’s SummaryWriter-compatible interfaces and on common training loop patterns. Every call to loss.backward(), every optimizer step, and every metric computed via PyTorch’s logging utilities is intercepted and forwarded to the active ClearML task. This architecture works seamlessly in script-based workflows where each training run is a separate Python process. In Jupyter notebooks — where the kernel is a persistent Python process that persists across cell executions — it creates a subtle trap.

# Jupyter notebook — Brightfield Data Sciences
# clearml 1.14.2, PyTorch 2.1.0, JupyterHub on GKE 1.27

# === CELL 1 (run October 8, 2026 at 14:02) ===
from clearml import Task
import torch

task_a = Task.init(
    project_name='churn_model_v3',
    task_name='churn_exp_046_lr001_dropout03'
)
# ClearML auto-patches PyTorch at the PROCESS level for this kernel.
# All subsequent PyTorch metric calls in this process → logged to task_a.

# ... training loop for experiment A (lr=0.001, dropout=0.3) ...
# Epochs 1–50 logged to churn_exp_046_lr001_dropout03.
# Final AUC: 0.887 at epoch 50.
task_a.close()  # Marks task as completed — does NOT remove the PyTorch auto-patch.


# === CELL 2 (run October 8, 2026 at 16:47, same kernel) ===
from clearml import Task
import torch

task_b = Task.init(
    project_name='churn_model_v3',
    task_name='churn_exp_047_lr01_dropout05'
)
# task_b is created correctly.
# BUT: PyTorch auto-patch from Cell 1 is still active in this process.
# The patch still points to task_a (churn_exp_046).
# Metrics from experiment B's training loop are logged to task_a, NOT task_b.

# ... training loop for experiment B (lr=0.01, dropout=0.5) ...
# Epochs (labeled 51–100 by the patch, since task_a already has 50 epochs)
# are appended to churn_exp_046_lr001_dropout03.
# churn_exp_047_lr01_dropout05 shows zero metrics (flat curves).
# Correct AUC for experiment B: 0.902 at epoch 50.
# Logged AUC for experiment B: not logged to task_b — invisible.

The data scientist compared experiment 046 (AUC 0.887, metrics logged correctly) with experiment 047 (flat curves, AUC appeared to be the pre-training baseline of 0.831). The conclusion seemed clear: lr=0.001, dropout=0.3 (experiment 046) outperformed lr=0.01, dropout=0.5 (experiment 047). The production model was deployed with experiment 046’s configuration on October 9, 2026.

The retainer developer identified the problem during the monthly audit by cross-checking epoch counts against configured max_epochs. Experiment 046 showed 100 epochs — twice the configured 50. The epoch timestamps showed a 2.5-hour gap at epoch 50, corresponding exactly to the gap between Cell 1’s completion and Cell 2’s execution in the notebook. The ClearML task metadata for both tasks was checked:

# Diagnosis using ClearML API
from clearml import Task

# Retrieve both tasks
task_a = Task.get_task(project_name='churn_model_v3', task_name='churn_exp_046_lr001_dropout03')
task_b = Task.get_task(project_name='churn_model_v3', task_name='churn_exp_047_lr01_dropout05')

# Check scalar metrics counts
print(f"task_a epochs logged: {len(task_a.get_logged_scalars()['epoch_auc']['train'])}")
# → 100 (expected 50 — double the configured max_epochs)

print(f"task_b epochs logged: {len(task_b.get_logged_scalars().get('epoch_auc', {}).get('train', []))}")
# → 0 (expected 50 — flat curves, no metrics)

# Check task timestamps
print(f"task_a started: {task_a.data.started}")  # → 2026-10-08T14:02:31
print(f"task_a completed: {task_a.data.completed}")  # → 2026-10-08T19:23:17
# task_a ran for 5h21m — expected ~2.5h for 50 epochs.
# Actual 2.5h gap in epochs at epoch 50 = experiment B's training time.
# task_b started: 2026-10-08T16:47:02, completed: 2026-10-08T19:20:54
# task_b ran for 2.5h but logged 0 metrics — all metrics went to task_a.

The fix was applied in two stages. For the immediate production issue, the retainer developer re-ran experiment 047 in a fresh Python process (not a notebook kernel, but a ClearML agent execution) and confirmed the correct AUC of 0.902. The production model was updated to use experiment 047’s hyperparameters. For the systematic issue, a kernel restart policy was documented and enforced:

# Fix Option 1: Kernel restart between experiments (immediate mitigation)
# Before running any Task.init() for a second experiment in the same notebook,
# use Kernel → Restart & Clear Output. This starts a fresh Python process.
# The new process has no ClearML auto-patches, so the next Task.init() is clean.

# Fix Option 2: Use ClearML agent queues (recommended for multi-experiment sessions)
# Instead of running training in Jupyter cells directly, submit each experiment
# as a ClearML agent task. Each agent execution is a separate process with
# its own auto-patching scope.

from clearml import Task

task = Task.init(
    project_name='churn_model_v3',
    task_name='churn_exp_047_lr01_dropout05',
    reuse_last_task_id=False
)
task.execute_remotely(queue_name='gpu-experiments', clone=False, exit_process=True)
# This exits the notebook kernel cell and submits the experiment to the agent queue.
# The agent runs each experiment in a fresh subprocess.
# No cross-task metric logging is possible.

# Fix Option 3: Explicit task assignment per logger call
# Use task.get_logger().report_scalar() instead of relying on auto-patching.
# Pass the task object explicitly to ensure metrics go to the correct task.
# More verbose but immune to auto-patch scope issues.

task_b = Task.init(project_name='churn_model_v3', task_name='churn_exp_047')
logger_b = task_b.get_logger()

for epoch in range(max_epochs):
    # ... training step ...
    logger_b.report_scalar('loss', 'train', value=train_loss, iteration=epoch)
    logger_b.report_scalar('auc', 'val', value=val_auc, iteration=epoch)
# Explicit logger calls are immune to auto-patch cross-task issues.

The retainer developer also ran a retrospective audit of the past three months of ClearML tasks in the churn_model_v3 project, scanning for the flat-curve signature (tasks with zero logged scalars despite being marked as Completed). Four additional experiments from August and September 2026 showed the same pattern — their metrics had been appended to predecessor tasks from the same notebook sessions. All four were re-run via ClearML agent queues to get accurate results. None of the four had been selected for production, so there was no additional model quality impact beyond the already-deployed incorrect configuration.

Total retainer engagement: 3.5 hours (0.5h monthly audit: flat curves on churn_exp_047 flagged; 1.0h trace cross-task logging: epoch count anomaly on churn_exp_046 confirms B metrics appended to A; 0.5h retrospective audit: 4 additional affected experiments identified; 0.5h kernel restart policy documentation; 0.5h ClearML agent queue setup for future multi-experiment sessions; 0.5h re-run churn_exp_047 via agent: AUC 0.902 confirmed). The corrected model was deployed on October 23, 2026.

Retainer billing was logged to HourTab with the clearml version, PyTorch version, JupyterHub environment details, and a link to the four affected task IDs. The client could verify the complete 3.5-hour work log at the shared HourTab URL without logging in.

Related retainer posts

• MLflow developer on retainer — experiment tracking and model registry retainer work
• Weights & Biases developer on retainer — W&B experiment tracking retainer work
• ZenML developer on retainer — ML pipeline orchestration retainer work