Comet ML developer on retainer: PyTorch DDP spawns 4 worker processes each calling Experiment() — 4 duplicate experiments per training run — AUC improvement appears 4× smaller in weekly report — Comet ML developer on monthly retainer

comet-ml 3.43.1 • PyTorch 2.1.0 • PyTorch DDP • 4×A100 • GKE 1.27 • comet_ml.Experiment() unguarded in training script • 4 duplicate experiments per run • metrics 4× over-represented • 0.048-point AUC gain appears as 0.021 • better model ranked below worse model 6 weeks • fix: LOCAL_RANK == 0 guard + disabled=True on other ranks • 3.0h retainer engagement

In the third week of October 2026, the ML team at Millbrook Predictive Systems Inc. ran a quarterly model selection review for their customer churn prediction pipeline. The review process compared AUC scores across all experiments logged in the churn_v4 Comet ML project over the previous six weeks. The team’s improved model variant, churn_v4_attention, had been running in shadow mode since early September alongside the production baseline churn_v4_baseline. The expected conclusion was straightforward: the attention model had consistently outperformed the baseline in offline evaluation. But the Comet ML project-level comparison showed the two models separated by only 0.021 AUC points, well within the team’s noise threshold for deployment decisions. The improved model was not promoted. The Comet ML retainer developer was asked to investigate two weeks later, and closed the incident in 3.0 hours.

Millbrook’s ML team uses PyTorch 2.1.0 with PyTorch DDP for all production training runs on a 4×A100 GKE node. Training is launched with torchrun --nproc_per_node=4, which spawns 4 worker processes with LOCAL_RANK values 0 through 3 on the same node. Each process runs an identical copy of the training script train_churn_v4.py from the beginning. The script was written for single-GPU training and later adapted to DDP by adding the DDP initialization boilerplate at the top. The Comet ML experiment initialization was left in place, unchanged, at the top of the script:

# train_churn_v4.py — original DDP adaptation (pre-incident)
# Millbrook Predictive Systems Inc., comet-ml 3.43.1, PyTorch 2.1.0

import os
import torch
import torch.distributed as dist
from comet_ml import Experiment  # imported at module level

# DDP initialization — added when script was adapted for multi-GPU
dist.init_process_group(backend="nccl")
local_rank = int(os.environ["LOCAL_RANK"])
torch.cuda.set_device(local_rank)

# Comet ML experiment — NOT guarded by LOCAL_RANK check
# Each of the 4 DDP processes calls this independently
experiment = Experiment(
    project_name="churn_v4",
    workspace="millbrook-ml",
)
experiment.add_tag("ddp-4gpu")
experiment.log_parameters({"model_variant": MODEL_VARIANT, "lr": LR, "epochs": EPOCHS})

model = ChurnAttentionModel().cuda()
model = torch.nn.parallel.DistributedDataParallel(model, device_ids=[local_rank])

# ... training loop ...
# DDP all-reduce synchronizes gradients; all 4 processes produce identical metrics
for epoch in range(EPOCHS):
    train_loss = train_one_epoch(model, train_loader, optimizer)
    val_auc = evaluate(model, val_loader)
    experiment.log_metric("train/loss", train_loss, epoch=epoch)
    experiment.log_metric("eval/auc", val_auc, epoch=epoch)  # 4× logged, identically

experiment.end()

Because DDP spawns 4 processes each running the full script, and because Experiment() is called outside any LOCAL_RANK guard, every training run created 4 independent Comet ML experiment entries. The 4 experiments were not linked to each other as sub-runs or replicas; they were full, top-level experiment entries in the churn_v4 project, each with the same tag (ddp-4gpu), the same logged parameters, and the same metric values. The identical metric values resulted from DDP’s all-reduce gradient synchronization: after the first all-reduce step, all 4 processes converged to the same model weights, and all 4 produced the same val_auc value on every subsequent evaluate() call.

Millbrook’s model selection process queried the Comet ML project API to retrieve all experiments and computed a weighted-average AUC for each model variant. The denominator for each model variant was the count of experiments tagged with that variant — which was 4 per training run rather than 1. For the baseline model (churn_v4_baseline), run across 4 DDP training jobs over 6 weeks, there were 16 experiment entries in Comet ML (4 experiments × 4 runs) all with AUC values around 0.831. For the attention model (churn_v4_attention), run across 4 DDP training jobs over the same period, there were 16 experiment entries all with AUC values around 0.879. When each model’s 16 experiments were averaged against each other, the result was correct: 0.831 vs. 0.879, a 0.048-point difference. The error was in the comparison method the team used: they computed a project-level average across all 32 experiments and then looked at how far each model variant’s experiment cluster deviated from the project mean. Because both clusters were inflated by the same 4× factor, the relative variance between them was correctly preserved — but when they then checked whether the improvement exceeded their deployment threshold of 0.03 AUC points, they applied the threshold to a per-variant average that was computed after normalizing against a project-mean that was itself inflated. The result was a diluted delta of 0.021 points, below the threshold.

The retainer developer confirmed the root cause in 0.5 hours. Querying the Comet ML API for experiment count per training run showed 4 experiments per torchrun job across all 8 runs in the project. The LOCAL_RANK environment variable was not logged as a Comet ML parameter in any experiment (it was present in the training environment but had not been passed to experiment.log_parameters()), so the 4 experiments per run were visually indistinguishable in the Comet ML UI. The fix had two parts. First, the training script was updated to guard experiment creation behind a LOCAL_RANK == 0 check and pass a no-op experiment to all other ranks:

# train_churn_v4.py — fixed DDP pattern
# Millbrook Predictive Systems Inc., comet-ml 3.43.1, PyTorch 2.1.0

import os
import torch
import torch.distributed as dist
from comet_ml import Experiment

dist.init_process_group(backend="nccl")
local_rank = int(os.environ["LOCAL_RANK"])
torch.cuda.set_device(local_rank)

# Create Comet ML experiment only on rank 0.
# All other ranks use a disabled experiment (no-op wrapper).
if local_rank == 0:
    experiment = Experiment(
        project_name="churn_v4",
        workspace="millbrook-ml",
    )
    experiment.add_tag("ddp-4gpu")
    experiment.log_parameter("local_rank", local_rank)
    experiment.log_parameters({"model_variant": MODEL_VARIANT, "lr": LR, "epochs": EPOCHS})
else:
    experiment = Experiment(
        project_name="churn_v4",
        workspace="millbrook-ml",
        disabled=True,   # no-op: logs nothing, creates no server-side entry
    )

model = ChurnAttentionModel().cuda()
model = torch.nn.parallel.DistributedDataParallel(model, device_ids=[local_rank])

for epoch in range(EPOCHS):
    train_loss = train_one_epoch(model, train_loader, optimizer)
    val_auc = evaluate(model, val_loader)
    # experiment.log_metric is a no-op on ranks 1-3 (disabled=True experiment)
    experiment.log_metric("train/loss", train_loss, epoch=epoch)
    experiment.log_metric("eval/auc", val_auc, epoch=epoch)

experiment.end()

The first post-fix training run produced exactly 1 Comet ML experiment entry for the full 4-GPU job, with the correct AUC of 0.879. The 36 extra-rank experiments from the 9 prior DDP runs (September through October, 4 experiments per run × 9 runs, minus 1 correct experiment per run) were archived in the Comet ML project. The retainer developer recomputed the model selection comparison using only the 1 canonical experiment per run: the attention model’s AUC was 0.879 vs. the baseline’s 0.831, a 0.048-point improvement. The attention model cleared the 0.03-point deployment threshold. It was promoted to production on October 23, 2026 — six weeks after it had first demonstrated that improvement in training.

The prevention measure was a CI grep check added to the GitLab CI pipeline that ran on every pull request touching train_*.py files. The check searched for comet_ml.Experiment( or from comet_ml import Experiment in any training script and flagged the file if no LOCAL_RANK guard was present within the same function scope. This check would have caught the original unguarded initialization before it reached production and accumulated 6 weeks of duplicated experiment data.

Total retainer engagement: 3.0 hours (0.5h root cause: Comet ML API query for experiments-per-run count, confirm 4 entries per torchrun job, identify missing LOCAL_RANK guard in training script; 0.5h impact assessment: identify which model selection decisions were affected, quantify the AUC delta dilution factor, confirm that the attention model’s actual AUC of 0.879 had consistently cleared the deployment threshold in all 4 training runs; 1.0h fix: add LOCAL_RANK guard with disabled=True for non-rank-0 processes, test with 2-GPU torchrun run confirming single experiment entry; 0.5h historical cleanup: archive 36 extra-rank experiment entries from September and October runs; 0.5h prevention: CI grep check for unguarded Experiment() calls, documentation update in team ML setup guide). Retainer billing was logged to HourTab with the comet-ml version, the DDP configuration, and the full fix applied, so the client could audit the complete 3.0-hour work log at the shared URL without logging in.

Related retainer posts

Track retainer hours without a client portal

HourTab turns your time-tracker CSV into a public URL your client bookmarks. No login. No portal. CSV in, URL out.

See pricing →