NNI developer on retainer: blanket try/except reports 0.0 for 30 failed trials — silent failures poison Hyperband surrogate — val_acc 0.72 — 3 weeks — NNI developer on monthly retainer

microsoft-nni 3.0.x • Python 3.11 • PyTorch 2.1.x • Hyperband advisor • max_trial_num=50, trial_concurrency=4 • trial script: blanket except Exception: nni.report_final_result(0.0) • ImageFolder path resolves to empty directory for 30/50 trials • FileNotFoundError silently caught • 30 trials report val_acc=0.0 with SUCCEEDED status • TPE surrogate trains on 30 poisoned 0.0 observations • Hyperband promotes 20 surviving trials • best val_acc 0.72 — developer concludes Hyperband does not work • fix: remove blanket except • NNI marks failed trials FAILED • excludes from surrogate • val_acc 0.72→0.84 • 1.5h retainer engagement

Eastfield NNI Search Ltd. used NNI 3.0.x with Hyperband advisor to search over learning rate, weight decay, and batch size for a PyTorch image classifier, with max_trial_num=50 and trial_concurrency=4. After all 50 trials completed, the best val_acc was 0.72 — identical to the untrained model’s random baseline on a 10-class task — and far below the team’s manual grid search result of 0.84. The developer spent three weeks checking NNI version, Hyperband max_resource and eta parameters, PyTorch version, and model architecture before engaging an NNI retainer developer.

The trial script and NNI configuration at the time of the engagement:

# Eastfield NNI Search Ltd. — NNI Hyperband HPO
# microsoft-nni 3.0.x, Python 3.11, PyTorch 2.1.x

# trial.py (the script NNI launches per trial):
import nni
import torch
import torch.nn as nn
from torchvision import datasets, transforms, models
from torch.utils.data import DataLoader
import os

def run_trial():
    params = nni.get_next_parameter()
    lr           = params['lr']
    weight_decay = params['weight_decay']
    batch_size   = int(params['batch_size'])

    # ← Problem: relative path that breaks in some worker environments
    data_dir = 'data/imagenet_subset'   # relative to CWD of trial worker
    # CWD of trial worker = NNI experiment dir (~/.nni/experiments//)
    # 'data/imagenet_subset' does not exist there
    # → ImageFolder raises FileNotFoundError
    transform = transforms.Compose([
        transforms.Resize(224),
        transforms.ToTensor(),
    ])
    train_ds = datasets.ImageFolder(os.path.join(data_dir, 'train'), transform)
    val_ds   = datasets.ImageFolder(os.path.join(data_dir, 'val'),   transform)

    train_loader = DataLoader(train_ds, batch_size=batch_size, shuffle=True,
                              num_workers=4, pin_memory=True)
    val_loader   = DataLoader(val_ds,   batch_size=batch_size, shuffle=False,
                              num_workers=4, pin_memory=True)

    model = models.resnet18(pretrained=False, num_classes=10).cuda()
    optimizer = torch.optim.Adam(model.parameters(), lr=lr, weight_decay=weight_decay)
    criterion = nn.CrossEntropyLoss()

    best_val_acc = 0.0
    for epoch in range(20):
        model.train()
        for X, y in train_loader:
            optimizer.zero_grad()
            loss = criterion(model(X.cuda()), y.cuda())
            loss.backward()
            optimizer.step()

        model.eval()
        correct = total = 0
        with torch.no_grad():
            for X, y in val_loader:
                pred = model(X.cuda()).argmax(dim=1)
                correct += (pred == y.cuda()).sum().item()
                total   += y.size(0)
        val_acc = correct / total
        nni.report_intermediate_result(val_acc)
        best_val_acc = max(best_val_acc, val_acc)

    nni.report_final_result(best_val_acc)

# ← Blanket try/except: the root cause
try:
    run_trial()
except Exception:
    # Developer added this to prevent NNI from hanging on unhandled exception.
    # Problem: ANY exception — including FileNotFoundError from the data path —
    # is silently caught. nni.report_final_result(0.0) is called, making NNI
    # believe the trial completed successfully with val_acc=0.0.
    nni.report_final_result(0.0)

Why silent 0.0 results poison the Hyperband surrogate

NNI’s Hyperband advisor operates by evaluating configurations at multiple fidelity levels (epochs) and promoting the top-performing fraction to higher brackets. It uses an internal TPE (Tree-structured Parzen Estimator) surrogate model to suggest which configurations to try next. The TPE surrogate learns from all completed trials — both the intermediate results (reported via nni.report_intermediate_result()) and the final results (reported via nni.report_final_result()).

When 30 of the 50 trials fail due to the broken data path and the blanket except calls nni.report_final_result(0.0), NNI records all 30 as SUCCEEDED with a final result of 0.0. The intermediate results for these 30 trials are either empty (if the exception was raised before any training epoch completed) or all 0.0 (if nni.report_intermediate_result() was called with 0.0 as well). Hyperband sees these 30 trials as the worst performers in every bracket and correctly does not promote them — but the TPE surrogate still trains on all 50 final results when suggesting new configurations.

In this case, the data loading bug happened to correlate with batch_size=128 in 22 of the 30 failing trials (larger batch sizes triggered an earlier code path that used the broken relative path). The TPE surrogate learned that batch_size=128 is associated with val_acc=0.0 and strongly discouraged it in future suggestions. All subsequent configurations used batch_size=32 or batch_size=64, even though the actual problem was the relative path, not the batch size. The 20 trials that did complete successfully happened to all use batch_size=32 and a narrow learning rate range, limiting the search to a small corner of the space.

# Diagnostic: check for silent failures in NNI experiment
# microsoft-nni 3.0.x, Python 3.11

import nni
from nni.experiment import Experiment

# Method 1: Check NNI experiment dashboard for SUCCEEDED trials with low results
# nni.get_all_experiments() returns experiment metadata
# For each experiment, nni.list_trial_jobs() returns trial status and results

import requests

exp_id = 'your_experiment_id'   # from nni.create_experiment() return value
# Or read from ~/.nni/experiments/

# Get all trial results via NNI REST API:
resp = requests.get(f'http://localhost:8080/api/v1/nni/trial-jobs')
trials = resp.json()

# Check for SUCCEEDED trials with suspiciously low final_metric:
print("Trials with SUCCEEDED status and val_acc near 0.0:")
zero_result_trials = []
for trial in trials:
    if trial['status'] == 'SUCCEEDED':
        final_metric = trial.get('finalMetricData')
        if final_metric:
            val_acc = float(final_metric[0]['data'])
            if val_acc < 0.1:   # below random baseline for 10-class task
                zero_result_trials.append(trial)
                print(f"  Trial {trial['id']}: status=SUCCEEDED, val_acc={val_acc:.4f}")

print(f"\nTotal: {len(zero_result_trials)} / {len(trials)} SUCCEEDED trials with val_acc < 0.1")
# Output:
# Trial abcdef: status=SUCCEEDED, val_acc=0.0000
# Trial bcdefg: status=SUCCEEDED, val_acc=0.0000
# ... (30 more)
# Total: 30 / 50 SUCCEEDED trials with val_acc < 0.1

# Method 2: Check trial stderr logs for suppressed exceptions
import os
import glob

nni_log_dir = os.path.expanduser(f'~/.nni/experiments/{exp_id}/log')
trial_stderr_logs = glob.glob(os.path.join(nni_log_dir, 'trial_*/stderr'))

print("\nTrials with suppressed exceptions in stderr:")
for log_path in trial_stderr_logs[:5]:
    with open(log_path) as f:
        stderr_content = f.read()
    if 'FileNotFoundError' in stderr_content or 'Error' in stderr_content:
        trial_id = os.path.basename(os.path.dirname(log_path))
        print(f"  Trial {trial_id}: exception in stderr (suppressed by try/except)")
# Output:
#   Trial abcdef: exception in stderr (suppressed by try/except)
#   Trial bcdefg: exception in stderr (suppressed by try/except)
# (30 trials show FileNotFoundError for data/imagenet_subset)

# Method 3: Check correlation between batch_size and 0.0 results
print("\nbatch_size distribution in 0.0 trials vs genuine trials:")
batch_size_zero = [t['hyperParameters']['batch_size'] for t in zero_result_trials]
genuine_trials  = [t for t in trials if t['status'] == 'SUCCEEDED'
                   and float(t['finalMetricData'][0]['data']) >= 0.1]
batch_size_genuine = [t['hyperParameters']['batch_size'] for t in genuine_trials]
print(f"  0.0 result trials: batch_size={set(batch_size_zero)}")
print(f"  Genuine trials:    batch_size={set(batch_size_genuine)}")
# Output:
#   0.0 result trials: batch_size={32, 64, 128}  ← not purely batch_size=128
#   Genuine trials:    batch_size={32, 64}        ← TPE avoided 128 incorrectly

The fix: remove the blanket except and fix the data path

The fix has two parts: first, fix the root cause (the relative data path that fails in NNI’s worker environment), and second, remove the blanket try/except from the trial script so that any future exceptions propagate to NNI and cause the trial to be marked as FAILED status. When a trial exits with an unhandled exception, NNI records it as FAILED, excludes the trial from the TPE surrogate and Hyperband bracket promotion calculations, and optionally retries it (depending on the maxTrialNum and retry configuration). With all 50 trials completing successfully on genuine data, the TPE surrogate trains on 50 real val_acc observations and the search proceeds normally.

# Fix: remove blanket try/except; fix absolute data path
# microsoft-nni 3.0.x, Eastfield NNI Search Ltd.

# trial.py (corrected version):
import nni
import torch
import torch.nn as nn
from torchvision import datasets, transforms, models
from torch.utils.data import DataLoader
import os

def run_trial():
    params = nni.get_next_parameter()
    lr           = params['lr']
    weight_decay = params['weight_decay']
    batch_size   = int(params['batch_size'])

    # ← Fix: absolute path — works regardless of trial worker CWD
    data_dir = '/home/ubuntu/projects/eastfield/data/imagenet_subset'
    # Verify data directory exists before proceeding:
    assert os.path.isdir(os.path.join(data_dir, 'train')), \
        f"Training data not found at {data_dir}/train"
    assert os.path.isdir(os.path.join(data_dir, 'val')), \
        f"Validation data not found at {data_dir}/val"

    transform = transforms.Compose([
        transforms.Resize(224),
        transforms.ToTensor(),
    ])
    train_ds = datasets.ImageFolder(os.path.join(data_dir, 'train'), transform)
    val_ds   = datasets.ImageFolder(os.path.join(data_dir, 'val'),   transform)

    train_loader = DataLoader(train_ds, batch_size=batch_size, shuffle=True,
                              num_workers=4, pin_memory=True)
    val_loader   = DataLoader(val_ds,   batch_size=batch_size, shuffle=False,
                              num_workers=4, pin_memory=True)

    model = models.resnet18(pretrained=False, num_classes=10).cuda()
    optimizer = torch.optim.Adam(model.parameters(), lr=lr, weight_decay=weight_decay)
    criterion = nn.CrossEntropyLoss()

    best_val_acc = 0.0
    for epoch in range(20):
        model.train()
        for X, y in train_loader:
            optimizer.zero_grad()
            loss = criterion(model(X.cuda()), y.cuda())
            loss.backward()
            optimizer.step()

        model.eval()
        correct = total = 0
        with torch.no_grad():
            for X, y in val_loader:
                pred = model(X.cuda()).argmax(dim=1)
                correct += (pred == y.cuda()).sum().item()
                total   += y.size(0)
        val_acc = correct / total
        nni.report_intermediate_result(val_acc)   # required for Hyperband bracket promotions
        best_val_acc = max(best_val_acc, val_acc)

    nni.report_final_result(best_val_acc)
    # ← No try/except. If any exception occurs, NNI records trial as FAILED status.
    # NNI excludes FAILED trials from TPE surrogate and Hyperband bracket logic.
    # The experiment continues with the remaining budget (retries are configurable).

# Remove the blanket except entirely — do NOT add it back
# If you need to log exceptions for debugging, use:
#   import traceback; traceback.print_exc()
# NNI captures trial stderr automatically; you can read it in the dashboard.

run_trial()

Validation

# Validation log — November 2026
# microsoft-nni 3.0.x, Python 3.11, PyTorch 2.1.x
# Hyperband HPO: ResNet-18 fine-tuning on 10-class imagenet subset, 50 trials

# ---- Before fix: blanket try/except + relative data path ----
# Trials with SUCCEEDED status:    50 / 50
# Trials with val_acc < 0.1:       30 / 50  (silent FileNotFoundError → 0.0)
# Genuine completed trials:        20 / 50
# TPE surrogate poisoned:          yes (30 observations at val_acc=0.0)
# batch_size=128 unfairly penalized: yes (22 of 30 failing trials used batch_size=128)
# Best val_acc:                     0.720

# ---- After fix: no try/except; absolute path ----
# Trials with SUCCEEDED status:    50 / 50  (all genuine)
# Trials with FAILED status:       0 / 50
# Trials with val_acc < 0.1:       0 / 50
# TPE surrogate:                   clean (50 genuine observations)
# batch_size distribution explored: {32: 18 trials, 64: 17 trials, 128: 15 trials}
# Best val_acc:                     0.843

# ---- Improvement ----
# val_acc: 0.720 → 0.843 (+17.1%)
# Genuine trials explored: 20 → 50 (+150%)

# ---- Best config found ----
# lr = 1.73e-04
# weight_decay = 2.14e-05
# batch_size = 128   ← note: batch_size=128 IS explored and works well

# ---- Hour breakdown ----
# Triage:     0.5h — 30/50 SUCCEEDED trials with val_acc=0.0;
#             trial stderr shows FileNotFoundError × 30; blanket except confirmed
# Root cause: 0.25h — relative path 'data/imagenet_subset' fails in NNI worker CWD;
#             blanket except suppresses error; nni.report_final_result(0.0) for 30 trials
# Fix:        0.25h — absolute path + assert; remove blanket except
# Validation: 0.5h — 50 SUCCEEDED with genuine results; val_acc 0.72→0.84; 1.5h total

Related retainer posts

Track retainer hours without a client portal

HourTab turns your time-tracker CSV into a public URL your client bookmarks. No login. No portal. CSV in, URL out.

See pricing →