Ax platform developer on retainer: placeholder metrics in attach_trial / complete_trial — BoTorch GP overfits to constant 0.75 — EI acquisition collapses to centroid — 50 wasted trials — 3 weeks — Ax platform developer on monthly retainer

ax-platform 0.3.x • Python 3.11 • BoTorch 0.9.x • PostgreSQL 15 • SQLAlchemy 2.0.x • 100-trial Bayesian optimization of CNN architecture • complete_trial(raw_data={'objective': 0.75}) hard-coded placeholder × 50 trials • GP training dataset: objective variance = 0 • EI acquisition collapses: all candidates score ~0 • get_next_trial() returns centroid parameters on every call • developer checks ax-platform version, BoTorch version, Python version — 3 weeks • fix: new experiment; re-run 50 trials with real val_loss from training • objective std 0.000→0.047 • EI explores search space correctly • best val_loss 0.75→0.68 • 1.5h retainer engagement

Southfield Ax Optimization Ltd. ran a 100-trial Bayesian optimization of a CNN architecture search space using ax-platform 0.3.x with a BoTorch 0.9.x GP surrogate and PostgreSQL 15 as the experiment store. Their workflow used Ax’s attach_trial() API to register manually launched training jobs rather than Ax’s built-in runner. To “get the loop moving” while waiting for training results, the developer called complete_trial(trial_index=idx, raw_data={'objective': 0.75}) with a hard-coded placeholder for each of the first 50 trials, intending to update the values later. After 50 trials, every call to get_next_trial() returned the identical parameter set — the centroid of the search space. The developer engaged an Ax retainer developer after three weeks.

The trial registration loop at the time of the engagement:

# Southfield Ax Optimization Ltd. — Ax platform BO study
# ax-platform 0.3.x, Python 3.11, BoTorch 0.9.x, PostgreSQL 15

from ax.service.ax_client import AxClient, ObjectiveProperties

ax_client = AxClient()
ax_client.create_experiment(
    name="cnn_arch_search",
    parameters=[
        {"name": "num_filters", "type": "range", "bounds": [16, 256], "value_type": "int"},
        {"name": "kernel_size", "type": "choice", "values": [3, 5, 7]},
        {"name": "dropout", "type": "range", "bounds": [0.0, 0.5]},
        {"name": "learning_rate", "type": "range", "bounds": [1e-5, 1e-2], "log_scale": True},
        {"name": "batch_size", "type": "choice", "values": [32, 64, 128, 256]},
        {"name": "weight_decay", "type": "range", "bounds": [1e-6, 1e-3], "log_scale": True},
    ],
    objectives={"objective": ObjectiveProperties(minimize=True)},
)

# Developer attached 50 trials from prior manual runs
for trial_params in prior_trial_configs:   # 50 manually-run param sets
    _, trial_index = ax_client.attach_trial(parameters=trial_params)

    # ← Problem: hard-coded placeholder metric for all 50 trials
    ax_client.complete_trial(
        trial_index=trial_index,
        raw_data={"objective": 0.75},   # ← intended to fill in real val_loss "later"
    )
    # Ax ServiceAPI does not support updating completed trial data.
    # These 50 observations are permanently stored with objective=0.75.

# After 50 trials: start BO phase
for _ in range(50):   # trials 51-100
    parameters, trial_index = ax_client.get_next_trial()
    # ← Every call returns the same parameters:
    # {'num_filters': 128, 'kernel_size': 5, 'dropout': 0.25,
    #  'learning_rate': 0.001, 'batch_size': 128, 'weight_decay': 1e-4}
    # (centroid of the search space)
    # Developer changes nothing — model still suggests centroid
    # Best trial after 100 runs: objective=0.75 (all placeholder values)

Why placeholder metrics collapse the BoTorch GP acquisition surface

Ax’s BoTorch backend fits a Gaussian Process surrogate on the trial observations to model the objective surface. The GP’s training data consists of parameter vectors X and their observed objective values y. When all 50 observations have y = 0.75, the GP’s marginal likelihood optimization drives the output scale hyperparameter toward zero and the noise variance toward zero, since there is no variance in y to explain. The resulting GP predicts 0.75 everywhere with near-zero uncertainty.

Expected Improvement (EI) is defined as EI(x) = E[max(f(x) − f*, 0)] where f* is the best observed value. With f* = 0.75 and the GP predicting mean 0.75 and variance 0 at every point, EI is identically zero across the entire search space. BoTorch’s acquisition optimizer (optimize_acqf) then returns whichever candidate is selected by the numerical initializer — in practice, the centroid of the parameter bounds. The developer sees get_next_trial() returning {'num_filters': 128, 'kernel_size': 5, 'dropout': 0.25, 'learning_rate': 0.001, 'batch_size': 128, 'weight_decay': 1e-4} for every trial and concludes the acquisition function is broken.

# Diagnostic: check objective variance before trusting GP optimization

import pandas as pd

# Retrieve trial data
df = ax_client.get_trials_data_df()
print(df[["trial_index", "objective_mean"]].to_string())
# Output:
#    trial_index  objective_mean
# 0            0            0.75
# 1            1            0.75
# 2            2            0.75
# ...
# 49          49            0.75
# (50 rows — all 0.75)

# Check objective variance
obj_std = df["objective_mean"].std()
obj_min = df["objective_mean"].min()
obj_max = df["objective_mean"].max()
print(f"Objective std:  {obj_std:.4f}")   # Output: 0.0000
print(f"Objective min:  {obj_min:.4f}")   # Output: 0.7500
print(f"Objective max:  {obj_max:.4f}")   # Output: 0.7500

# Zero std means placeholder data — GP cannot learn a response surface.
# Ax ServiceAPI: once complete_trial() is called, data cannot be updated.
# Any subsequent get_next_trial() call will use this degenerate GP.

# Confirm repeated suggestions:
for _ in range(3):
    params, idx = ax_client.get_next_trial()
    print(f"Trial {idx}: {params}")
    ax_client.abandon_trial(idx)
# Output:
#   Trial 51: {'num_filters': 128, 'kernel_size': 5, 'dropout': 0.25,
#              'learning_rate': 0.001, 'batch_size': 128, 'weight_decay': 1e-4}
#   Trial 52: {'num_filters': 128, 'kernel_size': 5, 'dropout': 0.25,
#              'learning_rate': 0.001, 'batch_size': 128, 'weight_decay': 1e-4}
#   Trial 53: {'num_filters': 128, 'kernel_size': 5, 'dropout': 0.25,
#              'learning_rate': 0.001, 'batch_size': 128, 'weight_decay': 1e-4}
# Identical parameters: EI surface flat → optimizer picks centroid

The fix: new experiment; re-run 50 trials with real metrics

Ax does not support modifying completed trial data. The remediation is to create a new experiment and re-run the 50 initialization trials with actual observed metrics. The developer already had the training job logs from the prior manual runs; the fix was to extract the real val_loss values from those logs and register them correctly. Once the 50 real observations were in place, the GP fitted a meaningful response surface and EI correctly identified regions of the search space that were better than the best observed value.

# Fix: new experiment + re-run attach_trial with real observed val_loss
# ax-platform 0.3.x, Southfield Ax Optimization Ltd.

from ax.service.ax_client import AxClient, ObjectiveProperties

# Create a fresh experiment — cannot recover contaminated data
ax_client_v2 = AxClient()
ax_client_v2.create_experiment(
    name="cnn_arch_search_v2",
    parameters=[
        {"name": "num_filters", "type": "range", "bounds": [16, 256], "value_type": "int"},
        {"name": "kernel_size", "type": "choice", "values": [3, 5, 7]},
        {"name": "dropout", "type": "range", "bounds": [0.0, 0.5]},
        {"name": "learning_rate", "type": "range", "bounds": [1e-5, 1e-2], "log_scale": True},
        {"name": "batch_size", "type": "choice", "values": [32, 64, 128, 256]},
        {"name": "weight_decay", "type": "range", "bounds": [1e-6, 1e-3], "log_scale": True},
    ],
    objectives={"objective": ObjectiveProperties(minimize=True)},
)

# Re-attach 50 prior trials with REAL observed val_loss
# (extracted from training logs of the prior manual runs)
prior_trial_results = [
    # (params_dict, real_val_loss)
    ({"num_filters": 32, "kernel_size": 3, "dropout": 0.1, "learning_rate": 3e-4, "batch_size": 64, "weight_decay": 1e-5}, 0.731),
    ({"num_filters": 64, "kernel_size": 5, "dropout": 0.3, "learning_rate": 1e-3, "batch_size": 128, "weight_decay": 5e-5}, 0.714),
    # ... (50 real (params, val_loss) pairs from prior training logs)
]

for trial_params, real_val_loss in prior_trial_results:
    _, trial_index = ax_client_v2.attach_trial(parameters=trial_params)
    ax_client_v2.complete_trial(
        trial_index=trial_index,
        raw_data={"objective": real_val_loss},   # ← real metric, not placeholder
    )

# Validate GP has diverse observations:
df2 = ax_client_v2.get_trials_data_df()
print(f"Objective std:  {df2['objective_mean'].std():.4f}")   # 0.0471
print(f"Objective min:  {df2['objective_mean'].min():.4f}")   # 0.6940
print(f"Objective max:  {df2['objective_mean'].max():.4f}")   # 0.7560

# BO phase: get_next_trial() now explores correctly
for _ in range(50):   # trials 51-100
    parameters, trial_index = ax_client_v2.get_next_trial()
    # Parameters vary across calls — EI identifies promising regions
    val_loss = run_training(parameters)   # real training run
    ax_client_v2.complete_trial(
        trial_index=trial_index,
        raw_data={"objective": val_loss},
    )

best_params, (best_vals, _), _ = ax_client_v2.get_best_parameters()
print(f"Best val_loss: {best_vals['objective']:.4f}")
# Before fix (placeholder data): best_val_loss = 0.750
# After fix  (real data):        best_val_loss = 0.682

Validation

# Validation log — November 2026
# ax-platform 0.3.x, Python 3.11, BoTorch 0.9.x, PostgreSQL 15
# 100-trial BO of CNN architecture (50 attach_trial init + 50 BO)

# ---- Before fix: placeholder metrics in complete_trial() ----
# ax_client.get_trials_data_df() objective std:  0.0000
# ax_client.get_trials_data_df() objective range: 0.75 → 0.75
# get_next_trial() output: same centroid parameters on every call
# Best val_loss after 100 trials: 0.750 (placeholder value)
# EI acquisition surface: flat, peak at search space centroid

# ---- After fix: real val_loss in complete_trial() ----
# ax_client_v2.get_trials_data_df() objective std:  0.0471
# ax_client_v2.get_trials_data_df() objective range: 0.694 → 0.756
# get_next_trial() output: diverse parameters exploring search space
# Best val_loss after 100 trials: 0.682
# EI acquisition surface: peaked in low-filter, low-dropout region

# ---- Best parameters found ----
# num_filters: 48
# kernel_size: 3
# dropout: 0.11
# learning_rate: 2.8e-4
# batch_size: 64
# weight_decay: 3.1e-6
# Best val_loss: 0.682 (vs 0.750 with placeholder data)

# ---- Hour breakdown ----
# Triage:     0.5h — repeated centroid suggestions; get_trials_data_df()
#             objective std = 0.000; identified complete_trial() placeholders
# Root cause: 0.25h — 50 trials registered with objective=0.75 placeholder;
#             Ax does not support updating completed trial data
# Fix:        0.25h — new experiment; 50 trials re-registered with real val_loss
# Validation: 0.5h — objective std 0.0471; get_next_trial() diverse;
#             best val_loss 0.682; 1.5h total

Related retainer posts

Track retainer hours without a client portal

HourTab turns your time-tracker CSV into a public URL your client bookmarks. No login. No portal. CSV in, URL out.

See pricing →