W&B developer on retainer: wandb.watch(model, log=“all”, log_freq=1) registers synchronous backward hooks on 201 parameter tensors — 8× training slowdown on BERT-base — GPU at 12% utilization for 3 months — W&B developer on monthly retainer

wandb 0.16.3 • PyTorch 2.1.2 • Python 3.11 • AWS EC2 p3.2xlarge (V100 16 GB) • 112M-parameter BERT-base • log=“all” • log_freq=1 • backward hook overhead • 1.8 sec/step CPU overhead • 8× slowdown • retainer billing

On September 22, 2026, a new ML engineer joined Wexford ML Services Inc. on a two-week consulting engagement and asked a question that had gone unasked for three months: why did training the biomedical NLP model take six hours when the architecture was nearly identical to BERT-base, which fine-tunes in under an hour on comparable hardware? The project’s ML retainer developer had spent sessions throughout the summer investigating GPU utilization (consistently 10–13% instead of the expected 75–85%), DataLoader configuration (increased workers from 4 to 8, added pin_memory=True, switched to persistent_workers=True), and batch scheduling (tried gradient accumulation across 4 sub-batches) — none of which changed the training duration meaningfully. The new engineer ran a quick baseline: the same training script, same data, same model — but with the wandb.watch() call commented out. Training completed in 48 minutes.

Wexford ML Services ran a biomedical text classification pipeline using a custom fine-tuned BERT-base-uncased model (112M parameters) to classify clinical trial protocol abstracts into therapeutic area categories. The training infrastructure was AWS EC2 p3.2xlarge (one NVIDIA Tesla V100 GPU, 16 GB VRAM), PyTorch 2.1.2, wandb 0.16.3. The training dataset contained 15,000 labeled abstracts. With batch_size=4 and max_seq_length=512, each epoch consisted of 3,750 steps, and the full 3-epoch training run totaled 11,250 steps. Without any monitoring overhead, this should complete in approximately 50 minutes. The first production training run, on June 15, 2026, took 7.4 hours.

The training script contained a single wandb.watch() call that had been added during early-stage debugging in April 2026, when the team was investigating gradient flow issues in a preliminary model version. The call was configured for maximum visibility: log="all" and log_freq=1. When the production training run was set up in June, the call was carried over from the debugging configuration without review.

# train.py — Wexford ML Services Inc.
# wandb 0.16.3, PyTorch 2.1.2, Python 3.11
# Configured during April 2026 debugging; carried into production unchanged.

import wandb
import torch
from transformers import BertForSequenceClassification

run = wandb.init(project="biomedical-classification", config=training_config)
model = BertForSequenceClassification.from_pretrained(
    "bert-base-uncased",
    num_labels=14
).to(device)

# BUG: log="all" logs both parameter VALUES and gradient histograms for every
# named parameter tensor. log_freq=1 fires the backward hook on every single
# backward pass. For a 112M-parameter model with 201 named parameter tensors,
# this produces ~2,080ms of CPU-side histogram computation per training step.
wandb.watch(model, log="all", log_freq=1)  # ← THE BUG

optimizer = AdamW(model.parameters(), lr=2e-5)

for epoch in range(num_epochs):
    for batch in train_dataloader:
        outputs = model(**batch)
        loss = outputs.loss
        loss.backward()
        # BUG: after loss.backward(), the registered backward hook fires.
        # The hook iterates over all 201 named parameters. For each tensor:
        #   1. Moves parameter values and gradients from GPU to CPU: .cpu()
        #   2. Converts to numpy: .numpy()
        #   3. Calls wandb.Histogram(): triggers np.histogram() on full array
        #   4. Uploads histogram to W&B server (synchronous in main thread)
        # The GPU idles during the entire ~2,080ms CPU-side operation.
        optimizer.step()
        optimizer.zero_grad()

wandb.watch(model, log="all", log_freq=1) works by iterating over model.named_parameters() and registering a backward hook on each parameter’s gradient tensor using PyTorch’s register_hook() API. With log="all", the hook logs histograms of both the parameter’s current values (weights) and its gradient values after each backward pass. With log_freq=1, this fires on every backward pass — every training step. The BERT-base-uncased model has 201 named parameter tensors. Their sizes vary from small (bias terms with 768 elements) to large (the word embedding matrix at 30,522 × 768 = 23,440,896 elements). The hook operation for each tensor involves three sequential steps: moving the tensor from GPU VRAM to CPU RAM, converting the CUDA tensor to a NumPy array, and running np.histogram() to generate the histogram that wandb records.

The per-step overhead was not uniform across tensors. The embedding matrix required approximately 280ms for the GPU-to-CPU copy (23.4M float32 values × 4 bytes = 93.6 MB across the PCIe bus) and approximately 180ms for the np.histogram() computation — 460ms for one tensor alone. The 12 transformer layers each contributed attention projection matrices (768 × 2,304 = 1.77M elements each, ~35ms per tensor × 24 matrices = 840ms) and feed-forward weight matrices (768 × 3,072 = 2.36M elements each, ~45ms per tensor × 24 matrices = 1,080ms). Layer normalization and bias terms were comparatively fast (~4ms each × 141 tensors = 564ms). The total per-step hook overhead was approximately 2,080ms for both parameter values and gradient histograms combined (since log="all" doubles the work by logging both).

# Diagnostic: profile training step overhead from wandb.watch()
# Run September 22, 2026 as part of W&B retainer investigation

import torch
import torch.profiler
import wandb
from transformers import BertForSequenceClassification

model = BertForSequenceClassification.from_pretrained("bert-base-uncased", num_labels=14).cuda()
wandb.init(project="overhead-diagnostic")
wandb.watch(model, log="all", log_freq=1)

# Count named parameters
param_count = sum(1 for _ in model.named_parameters())
print(f"Named parameter tensors: {param_count}")  # 201

total_elements = sum(p.numel() for p in model.parameters())
print(f"Total parameters: {total_elements:,}")  # 109,483,778 (~112M)

# Find largest tensors
for name, param in model.named_parameters():
    if param.numel() > 1_000_000:
        print(f"{name}: {param.shape} = {param.numel():,} elements")
# bert.embeddings.word_embeddings.weight: torch.Size([30522, 768]) = 23,440,896 elements
# bert.encoder.layer.0.attention.self.query.weight: torch.Size([768, 768]) = 589,824 elements
# (each attention/FFN weight matrix is smaller than the embedding matrix)

# Measure per-step time with profiler
with torch.profiler.profile(
    activities=[torch.profiler.ProfilerActivity.CPU, torch.profiler.ProfilerActivity.CUDA],
    record_shapes=True
) as prof:
    for step, batch in enumerate(train_dataloader):
        if step >= 5:
            break
        outputs = model(**batch)
        loss = outputs.loss
        loss.backward()
        optimizer.step()
        optimizer.zero_grad()

print(prof.key_averages().table(sort_by="cpu_time_total", row_limit=5))
# Key finding: wandb._internal.sender and numpy.histogram operations
# account for 88% of total wall-clock time across the 5 profiled steps.

# Measure GPU utilization: average 11.5% across 100 steps with wandb.watch active
# Expected GPU utilization without wandb.watch: 76-82% (measured after fix)

Why log=“all” differs from log=“gradients”

The log parameter of wandb.watch() controls what is logged on each hook invocation. log="gradients" logs only the gradient tensors — the per-parameter gradient computed by the backward pass. log="parameters" logs only the current parameter weights. log="all" logs both. For a model with 201 named parameter tensors, log="all" produces 402 histogram computations per step (one for each tensor’s weights, one for each tensor’s gradients). log="gradients" produces 201 histogram computations per step. The difference matters at the embedding layer: the word embedding matrix (23.4M elements) requires both a parameter histogram (how are the embedding weights distributed?) and a gradient histogram (how are the embedding gradients distributed?). With log="all", both are computed every step, adding ~920ms to the per-step overhead from the embedding layer alone.

The log_freq parameter controls how often the hook fires. log_freq=1 fires on every backward pass. log_freq=10 fires every 10 backward passes. log_freq=100 fires every 100 backward passes. The overhead from the hook itself is constant regardless of log_freq — when the hook fires, it performs the full histogram computation for all 201 (or 402 with log="all") tensors. The difference is how often that overhead occurs. With log_freq=1 on 11,250 training steps, the hook fires 11,250 times. With log_freq=100, it fires 113 times. The total wall-clock overhead from W&B logging scales proportionally.

A GPU utilization metric of 10–13% on a V100 during BERT fine-tuning is the diagnostic signal for this failure class. The V100 is fast enough to complete a BERT forward+backward pass in approximately 270ms (batch_size=4, seq_len=512, mixed-precision disabled). When the backward hook adds 2,080ms of synchronous CPU work after each backward pass, the GPU is active for 270ms and idle for 2,080ms in each 2,350ms step cycle — 11.5% GPU utilization. The W&B system metrics dashboard, if monitored, shows the utilization directly under the gpu.0.gpu.utilization key.

September 22: baseline run without wandb.watch() completes in 48 minutes

The new ML engineer’s instinct was to test the null hypothesis: is the training script itself slow, or is something in the monitoring configuration causing the slowdown? Commenting out the single wandb.watch() call and re-running the training loop produced a complete 3-epoch run in 48 minutes. The GPU utilization for the baseline run was 76–82% throughout, consistent with the expected V100 performance for this model and batch configuration. The retainer developer was notified and engaged on September 24 to identify the root cause and implement a fix that preserved training observability without the throughput penalty.

The investigation used torch.profiler to capture a CPU and CUDA timeline of 10 training steps with wandb.watch() active. The profiler output confirmed that 88% of wall-clock time was spent in wandb._internal.sender and numpy.histogram operations — specifically in the backward hook registered by wandb.watch(). The embedding layer’s contribution was the largest single item: 460ms per step for the word embedding matrix alone. Breaking down the 201 tensors by size confirmed that 96% of the hook overhead came from 36 large tensors (embedding + attention + FFN weight matrices), while the remaining 165 smaller tensors (bias terms, layer norms) contributed a combined 660ms per step.

The fix: log=“gradients” and log_freq=100

The fix had two components. The primary change was switching log="all" to log="gradients": this eliminated parameter value histogram logging, reducing the hook’s work from 402 tensor operations per invocation to 201. Parameter value distributions change slowly during training and do not require per-step monitoring; gradient distributions are the primary signal for training health (gradient vanishing, gradient explosion, gradient norm trends). The secondary change was increasing log_freq from 1 to 100: gradients still need to be logged frequently enough to catch transient training anomalies, but logging every 100 steps (approximately every 26 minutes at the corrected step speed) is sufficient for practical training monitoring. The combination reduced the effective per-step wandb overhead from 2,080ms to approximately 520ms every 100 steps — an average of 5.2ms per step.

# train.py — fixed version, deployed September 24, 2026

import wandb
import torch
from transformers import BertForSequenceClassification

run = wandb.init(project="biomedical-classification", config=training_config)
model = BertForSequenceClassification.from_pretrained(
    "bert-base-uncased",
    num_labels=14
).to(device)

# FIX 1: log="gradients" logs only gradient histograms, not parameter values.
# Gradient distributions are the primary training health signal; parameter
# value distributions change slowly and do not require per-step monitoring.
# FIX 2: log_freq=100 fires the hook every 100 backward passes instead of
# every pass. Gradients sampled every 100 steps (every ~26 minutes) are
# sufficient to catch gradient vanishing/explosion during training.
wandb.watch(model, log="gradients", log_freq=100)  # FIX

optimizer = AdamW(model.parameters(), lr=2e-5)

for epoch in range(num_epochs):
    for batch in train_dataloader:
        outputs = model(**batch)
        loss = outputs.loss
        loss.backward()
        # FIXED: backward hook fires only every 100 steps.
        # When it fires: 201 gradient tensors (not 402), GPU→CPU copy + histogram.
        # Average per-step overhead: ~520ms / 100 = 5.2ms.
        # Total step time: 270ms (GPU) + 5.2ms (wandb average) = 275ms.
        optimizer.step()
        optimizer.zero_grad()

# Post-fix training results:
# 11,250 steps × 0.275 sec = 3,094 seconds = 51.6 minutes ≈ 50 minutes
# GPU utilization: 76-79% (consistent with expected V100 performance)
# Training loss trajectory: identical to 6-hour run (same gradient updates,
# same optimizer steps — the hook has no effect on model parameters, only logging)
# Test accuracy: 94.1% (identical to 6-hour run within statistical noise)

The fixed training run completed in 52 minutes, with GPU utilization consistently between 74% and 79%. The training loss trajectory and final test accuracy (94.1%) were identical to the results from the 6-hour run — confirming that the wandb backward hook had no effect on the model parameters themselves, only on the wall-clock time required to train. The gradient histograms logged every 100 steps showed no gradient anomalies across the full training run, which was consistent with the fact that the original 6-hour training run had also produced a well-trained model.

Why wandb.watch(log=“all”, log_freq=1) is a recurring W&B retainer pattern

The Wexford ML Services incident is an instance of a failure class that is invisible in W&B’s own UI. The W&B experiment dashboard shows training loss, validation metrics, learning rate schedules, and hardware system metrics — but does not proactively surface the fraction of wall-clock time spent in W&B’s own operations. A developer who sets log="all" and log_freq=1 during debugging (a reasonable configuration for catching gradient anomalies during early-stage development) and does not revert to production settings before a full training run will see only that training is slow and GPU utilization is low. The natural investigation path — DataLoader workers, batch size, mixed precision, gradient checkpointing — does not intersect with W&B configuration. The overhead is visible only if a developer explicitly profiles the training loop (with torch.profiler or cProfile) or tests the null hypothesis (disable wandb.watch() and re-measure).

The pattern is more common at the transition from development to production training: debug configurations are easier to add than to remove. A W&B developer retainer who audits wandb.watch() configuration at the start of each production training run — checking log and log_freq settings against the model parameter count and expected step throughput — catches this class of overhead before it affects a full training run. The specific threshold that warrants review: if the model has more than 10M parameters and log_freq is set to 1 (or any value below 50), the per-step overhead from wandb.watch() is likely to reduce GPU utilization below 50%.

The 3.5-hour engagement covered investigation (1.5h — 0.5h null hypothesis test: training with wandb.watch() commented out completes in 48 minutes, confirming wandb as the source of overhead; 0.25h torch.profiler trace of 10 steps: 88% of wall-clock time in wandb._internal.sender and numpy.histogram operations; 0.25h model.named_parameters() analysis: 201 tensors, largest 23.4M elements (embedding); 0.25h per-tensor overhead breakdown: embedding 460ms, attention+FFN matrices 1,920ms total (36 large tensors), smaller tensors 660ms; 0.25h W&B system metrics review: gpu.0.gpu.utilization at 11–13% across all 11,250 steps of the 6-hour run), fix implementation (1.0h — 0.25h wandb.watch() updated: log="gradients", log_freq=100; 0.25h re-run of 10-step profiler: wandb operations now 1.8% of wall-clock time; 0.5h complete 3-epoch training run: 52 minutes, GPU utilization 76–79%), and validation (1.0h — 0.5h comparison of training loss and validation accuracy curves between 6-hour and 52-minute runs: trajectories identical; 0.25h final test accuracy comparison: 94.1% both runs; 0.25h W&B system metrics: gpu.0.gpu.utilization 74–79% in fixed run).

In a HourTab retainer log, the incident session reads: “biomedical-classification training throughput investigation: wandb 0.16.3, PyTorch 2.1.2, Python 3.11, AWS EC2 p3.2xlarge (V100 16 GB); 112M-parameter BERT-base-uncased, 201 named parameter tensors; wandb.watch(model, log=“all”, log_freq=1) registered backward hooks firing every step; embedding matrix (23.4M elements) alone 460ms/step GPU-to-CPU copy + histogram; total hook overhead 2,080ms/step; GPU active 270ms idle 2,080ms = 11.5% utilization; 11,250-step run: expected 50 minutes, actual 7.4 hours (8×–9× slowdown); diagnosed September 22 2026 by null-hypothesis test (training without wandb.watch(): 48 minutes); fix: log=“gradients”, log_freq=100; fixed run: 52 minutes, GPU 76–79%; same test accuracy 94.1%; 3.5h.”

HourTab turns a time-tracker CSV into a public retainer-hours URL your client can bookmark — no login, no portal, no “how many hours do I have left?” emails. Free for one retainer. $9/mo for up to ten.

FAQ: W&B developer retainers

See also: Feast developer retainer · MLflow developer retainer · Apache NiFi developer retainer