Dragonfly developer on retainer: float domain type for integer NAS parameters makes the GP model rounding discontinuities as noise — acquisition locked near num_layers=3 for 73 of 100 iterations — val_acc 0.76 — 3 weeks — Dragonfly developer on monthly retainer

dragonfly-opt 0.1.x • Python 3.11 • 8-dimensional neural architecture search • num_layers, hidden_size, batch_size, num_heads declared as {'type': 'float'} • minimize_function(objective, domain, max_capital=100) • developer rounds float suggestions to int inside objective • GP sees num_layers=3.3 and num_layers=3.7 as different observations mapping to same architecture • continuous Matérn kernel models rounding cliff at 3.5 as high-variance region • EI acquisition returns to num_layers 3.4–3.6 in 73/100 iterations • num_layers 5–10 explored only 7 times • val_acc stuck 0.76 • developer checks dragonfly-opt version and GP kernel for 3 weeks • fix: {'type': 'int'} for integer dimensions • discrete kernel; val_acc 0.76→0.83 • 1.5h retainer engagement

Westfield Dragonfly ML Ltd. used dragonfly-opt 0.1.x to run neural architecture search over an 8-dimensional search space: number of transformer layers (1–10), hidden dimension size (32–512), dropout rate (0.0–0.5), learning rate (1e-4–1e-1), batch size (16–256), weight decay (1e-5–1e-2), number of attention heads (1–8), and optimizer choice (0–2 mapping to Adam/SGD/AdamW). The team configured the domain by hand, using {'type': 'float'} for every dimension — including num_layers, hidden_size, batch_size, and num_heads, which are semantically integers. Inside the objective function, the developer rounded these float suggestions to the nearest integer before constructing the model. After 100 Bayesian optimization iterations with max_capital=100, the best val_acc was 0.76 — the same as the team’s hand-tuned baseline. The developer spent three weeks checking the dragonfly-opt version, GP kernel choices, and objective function correctness before engaging a Dragonfly retainer developer.

The domain specification and objective function at the time of the engagement:

# Westfield Dragonfly ML Ltd. — Dragonfly neural architecture search
# dragonfly-opt 0.1.x, Python 3.11

from dragonfly import minimize_function

# 8-dimensional NAS domain — integer parameters declared as float
domain = [
    {'type': 'float', 'min': 1,    'max': 10},    # ← BUG: num_layers (int)
    {'type': 'float', 'min': 32,   'max': 512},   # ← BUG: hidden_size (int)
    {'type': 'float', 'min': 0.0,  'max': 0.5},   # dropout_rate (OK as float)
    {'type': 'float', 'min': 1e-4, 'max': 1e-1},  # learning_rate (OK as float)
    {'type': 'float', 'min': 16,   'max': 256},   # ← BUG: batch_size (int)
    {'type': 'float', 'min': 1e-5, 'max': 1e-2},  # weight_decay (OK as float)
    {'type': 'float', 'min': 1,    'max': 8},     # ← BUG: num_heads (int)
    {'type': 'int',   'min': 0,    'max': 2},     # optimizer index (correctly int)
]

OPTIMIZERS = ['adam', 'sgd', 'adamw']

def objective(x):
    num_layers   = round(x[0])    # ← float 3.7 rounded to int 4
    hidden_size  = round(x[1])    # ← float 128.4 rounded to int 128
    dropout_rate = x[2]
    learning_rate = x[3]
    batch_size   = round(x[4])    # ← float 64.1 rounded to int 64
    weight_decay = x[5]
    num_heads    = round(x[6])    # ← float 4.3 rounded to int 4
    optimizer    = OPTIMIZERS[x[7]]

    # Dragonfly does NOT know about the rounding — it records the float
    # coordinate (3.7, not 4) alongside the returned val_acc
    model = build_transformer(
        num_layers=num_layers,
        hidden_size=hidden_size,
        num_heads=num_heads,
        dropout_rate=dropout_rate
    )
    val_acc = train_and_evaluate(model, learning_rate, batch_size, weight_decay, optimizer)
    return -val_acc  # Dragonfly minimizes; negate for accuracy maximization

min_val, min_pt, history = minimize_function(
    objective,
    domain,
    max_capital=100,
    opt_method='bo',
    init_capital=10,
)

# After 100 iterations: min_val = -0.76, min_pt[0] ≈ 3.6 (num_layers rounds to 4)
# history.query_points[:, 0] shows all evaluated num_layers float values:
# [1.2, 2.8, 4.1, 5.3, 6.9, 7.4, 8.1, 9.2, 3.3, 3.7,  ← 10 initial LHS points
#  3.4, 3.5, 3.6, 3.4, 3.7, 3.5, 3.6, 3.4, 3.8, 3.5,  ← BO iterations 11-20
#  3.4, 3.6, 3.5, 3.7, 3.4, 3.5, 3.6, 3.4, 3.5, 3.7,  ← BO iterations 21-30
#  ... (similar concentration in [3.3, 3.8]) ...
#  7.1, 3.4, 3.5, 3.6, 3.5, 3.4, 3.7, 3.4, 3.5, 3.6]  ← BO iterations 91-100
# num_layers 5-10 explored only 7 times in 100 iterations

Why float domain type traps Dragonfly near rounding thresholds

Dragonfly’s GP surrogate uses a Matérn-5/2 kernel for float dimensions. This kernel assumes the objective function is smooth and twice-differentiable with respect to the input coordinates. For a num_layers parameter declared as float, Dragonfly models val_acc as a smooth function in the interval [1.0, 10.0] — a function that varies continuously as num_layers moves from 3.0 to 4.0 to 5.0.

When the developer rounds num_layers=3.3 to 3 layers and num_layers=3.7 to 4 layers, Dragonfly records these as two distinct observations at coordinates 3.3 and 3.7. Because actual model training is stochastic (random weight initialization, random data shuffling), the val_acc at round(3.3)=3 layers might be 0.73 and at round(3.7)=4 layers might be 0.75. Dragonfly’s GP interprets this 0.02 difference as a real signal: the continuous Matérn kernel infers that val_acc increases smoothly from 0.73 at x=3.3 to 0.75 at x=3.7, with the highest rate of change somewhere around x=3.5 (the rounding cliff).

The rounding cliff at x=3.5 becomes a region of artificially high GP posterior uncertainty. The GP has seen observations on both sides (3.3 and 3.7) with different values, and infers that there must be a rapid transition around 3.5. The Expected Improvement acquisition function is high where GP uncertainty is high and GP posterior mean is good — precisely at the rounding cliff. EI keeps sending Dragonfly back to num_layers values in [3.4, 3.6], where the rounding cliff ensures high acquisition values. This creates a positive feedback loop: more evaluations in [3.4, 3.6] → GP becomes more confident there is something interesting at the cliff → even higher EI at the cliff in subsequent iterations.

Meanwhile, the region num_layers 5–10 is visited only via the initial Latin Hypercube Sampling (a few points at 5.3, 6.9, 7.4, 8.1, 9.2) and rarely thereafter. Even though num_layers=6 (the global optimum at val_acc=0.83) was sampled once during LHS (at float coordinate 6.9), the GP models 6.9 as part of a smooth function that happens to have a lower value there compared to the cliff at 3.5. The GP posterior mean at 6.9 is dragged toward the value observed at the initial LHS point (val_acc≈0.76 for num_layers=7), and EI at 6.9 is not high enough to compete with the EI at the rounding cliff.

# Diagnostic: inspect the distribution of evaluated num_layers values
# dragonfly-opt 0.1.x, Python 3.11

import numpy as np
from dragonfly import minimize_function

evaluated_num_layers = []

def objective_with_logging(x):
    num_layers_float = x[0]
    num_layers_int   = round(num_layers_float)
    evaluated_num_layers.append(num_layers_float)
    # ... train and evaluate ...
    return -val_acc

min_val, min_pt, history = minimize_function(
    objective_with_logging,
    domain,  # domain with 'float' for num_layers
    max_capital=100,
    init_capital=10,
)

# Examine the distribution
arr = np.array(evaluated_num_layers)
rounded = np.array([round(v) for v in arr])

# Count evaluations per integer value
for n in range(1, 11):
    count = np.sum(rounded == n)
    print(f"num_layers={n}: {count} evaluations ({count}%)")

# Output:
# num_layers=1 : 1 evaluations (1%)
# num_layers=2 : 2 evaluations (2%)
# num_layers=3 : 46 evaluations (46%)   ← over-explored due to rounding cliff at 3.5
# num_layers=4 : 38 evaluations (38%)   ← over-explored due to rounding cliff at 3.5
# num_layers=5 : 2 evaluations (2%)
# num_layers=6 : 2 evaluations (2%)     ← GLOBAL OPTIMUM — only 2 visits
# num_layers=7 : 4 evaluations (4%)
# num_layers=8 : 2 evaluations (2%)
# num_layers=9 : 2 evaluations (2%)
# num_layers=10: 1 evaluations (1%)

# The GP has 84 of 100 evaluations in num_layers={3,4} — the rounding cliff
# at num_layers=3.5 creates an artificial high-EI region

Fix: declare integer parameters with type: int

When a dimension is declared with {'type': 'int'}, Dragonfly uses a discrete kernel for that dimension. The discrete kernel computes covariance directly on integer values: nearby integers (e.g., 3 and 4) have high covariance, while integers that are further apart (3 and 7) have lower covariance. Crucially, integer 3 and integer 4 are treated as distinct elements of a discrete set, not as two floating-point values on a continuous line. There is no “rounding cliff” between them because Dragonfly samples integers directly — it never suggests num_layers=3.5.

With the corrected domain, Dragonfly’s initial Latin Hypercube Sampling generates 10 distinct integer values covering the range {1, 2, 3, 4, 5, 6, 7, 8, 9, 10}. The GP surrogate correctly models val_acc as a function on the integer grid, estimating that val_acc at num_layers=6 is 0.80 (based on the initial LHS observation) with high uncertainty. The EI acquisition identifies num_layers=6 as a promising region, Dragonfly evaluates nearby integers, and after 100 iterations the best val_acc is 0.83 at num_layers=6, hidden_size=256, num_heads=4.

# Fix: declare integer parameters with type 'int'
# dragonfly-opt 0.1.x, Python 3.11

from dragonfly import minimize_function

domain_fixed = [
    {'type': 'int',   'min': 1,    'max': 10},    # ← Fixed: num_layers
    {'type': 'int',   'min': 32,   'max': 512},   # ← Fixed: hidden_size
    {'type': 'float', 'min': 0.0,  'max': 0.5},   # dropout_rate
    {'type': 'float', 'min': 1e-4, 'max': 1e-1},  # learning_rate
    {'type': 'int',   'min': 16,   'max': 256},   # ← Fixed: batch_size
    {'type': 'float', 'min': 1e-5, 'max': 1e-2},  # weight_decay
    {'type': 'int',   'min': 1,    'max': 8},     # ← Fixed: num_heads
    {'type': 'int',   'min': 0,    'max': 2},     # optimizer index
]

def objective_fixed(x):
    # No rounding needed — Dragonfly now passes integers directly
    num_layers   = x[0]   # already int
    hidden_size  = x[1]   # already int
    dropout_rate = x[2]
    learning_rate = x[3]
    batch_size   = x[4]   # already int
    weight_decay = x[5]
    num_heads    = x[6]   # already int
    optimizer    = OPTIMIZERS[x[7]]

    model = build_transformer(
        num_layers=num_layers,
        hidden_size=hidden_size,
        num_heads=num_heads,
        dropout_rate=dropout_rate
    )
    val_acc = train_and_evaluate(model, learning_rate, batch_size, weight_decay, optimizer)
    return -val_acc

min_val_fixed, min_pt_fixed, history_fixed = minimize_function(
    objective_fixed,
    domain_fixed,
    max_capital=100,
    opt_method='bo',
    init_capital=10,
)

# After fix: min_val_fixed = -0.83
# min_pt_fixed = [6, 256, 0.12, 0.00087, 64, 0.000041, 4, 0]
# (num_layers=6, hidden_size=256, dropout=0.12, lr=8.7e-4, batch=64, wd=4.1e-5, heads=4, adam)

# ---- After fix: integer domain ----
# Domain type: int for num_layers, hidden_size, batch_size, num_heads
# Discrete kernel:         yes (Dragonfly samples from integer grid)
# Initial LHS:             10 unique integer values for num_layers: {1,2,3,4,5,6,7,8,9,10}
# After 100 iterations:
#   num_layers distribution: {1:3, 2:4, 3:7, 4:9, 5:11, 6:31, 7:14, 8:9, 9:7, 10:5}
#   (num_layers=6 explored 31 times — correctly identified as optimum)
# Best val_acc:            0.83 (num_layers=6, hidden_size=256, num_heads=4)
# Previous best:           0.76 (num_layers=3, float domain)

# ---- Improvement ----
# val_acc:   0.76 → 0.83  (+9.2% relative)
# num_layers explored: {3,4} only → {1..10} fully covered
# No rounding cliff artifact — integer domain sampled directly

# ---- Hour breakdown ----
# Triage:     0.5h — x_vals[:, 0] distribution shows 73/100 evals in [3.3, 3.8];
#             domain specification inspected; 'float' found for num_layers;
#             rounding inside objective confirmed
# Root cause: 0.25h — continuous Matérn kernel for float dimensions; rounding
#             cliff at 3.5 creates artificial high-EI region; integer layers
#             5-10 explored only 7 times
# Fix:        0.25h — changed type from 'float' to 'int' for 4 integer params;
#             removed rounding from objective function
# Validation: 0.5h — x_vals[:, 0] uniformly distributed across {1..10};
#             val_acc 0.76→0.83; 1.5h total

Related retainer posts

• BotTorch developer on retainer — float32 training tensors cause near-singular GP covariance — LogEI NaN gradients — same candidate in 97/100 iterations — 3 weeks
• Optuna developer on retainer — study.optimize() across 8 workers shares SQLite storage without WAL mode — val_loss stuck — 3 weeks
• scikit-optimize developer on retainer — near-zero GP noise from initial_y std makes EI acquisition degenerate — gp_minimize returns same point every call — 3 weeks