Alibi Detect developer on retainer: MMDDrift on 50-row production windows produces 42% false positive rate — real data pipeline drift dismissed as noise — model AUC degrades from 0.823 to 0.741 over three weeks undetected — Alibi Detect developer on monthly retainer

alibi-detect 0.11.4 • Python 3.10 • MMDDrift tabular drift detector • 5,000-row reference • 50-row production windows • n_permutations=100 default • 42% empirical false positive rate • alert fatigue by week 2 • NaN-to-zero imputation drift October 4 2026 • dismissed as false positive • AUC 0.823→0.741 over 3 weeks • fix: 1,000-row window accumulation + n_permutations=1,000 • 3.0h retainer engagement

In late October 2026, the data science team at Thornberry B2B Analytics Inc. engaged the Alibi Detect retainer developer to diagnose a model performance degradation that had gone undetected for three weeks. The churn prediction model’s AUC had declined from 0.823 to 0.741 between early October and late October without any alert from the MMDDrift monitoring pipeline. When the retainer developer reviewed the alert logs, the picture was the opposite of silence: the MMDDrift detector had been firing alerts in 42% of all 15-minute monitoring windows since the pipeline launched in August. The team had started treating every alert as a noise event after the first week. The root cause was a MMDDrift detector configured to run on 50-row production windows — a window size so small that the Maximum Mean Discrepancy test was statistically underpowered, producing false positive rates far above the nominal 5% significance level. The total engagement was 3.0 hours.

Thornberry B2B Analytics runs a churn prediction model for a B2B SaaS product with a low-volume scoring profile: 200 predictions per hour, arriving in batches of 50 every 15 minutes. The model scores each batch and publishes churn probabilities to the CRM dashboard. In August 2026, the team deployed Alibi Detect to add drift monitoring. The setup followed a standard pattern for tabular drift detection:

# monitoring/setup_detector.py — Thornberry B2B Analytics, alibi-detect 0.11.4
# Fitted August 2026; reference: 5,000 rows from Jan–Jun 2026 scoring history
from alibi_detect.cd import MMDDrift
import numpy as np
import joblib

reference_data = load_reference_data("2026-01-01", "2026-06-30")  # 5,000 rows
X_ref = reference_data[FEATURE_COLS].values.astype(np.float32)

# Standard instantiation — n_permutations defaults to 100
detector = MMDDrift(
    X_ref,
    p_val=0.05,          # significance threshold
    backend="pytorch",
    # n_permutations=100 implicitly — not overridden
)

joblib.dump(detector, "monitoring/mmd_detector.pkl")
print(f"Fitted MMDDrift on {len(X_ref)} reference rows")

The production monitoring job ran every 15 minutes, loading the 50 most recent inference rows and passing them to the fitted detector:

# monitoring/run_detector.py — runs every 15 minutes via cron
from alibi_detect.cd import MMDDrift
import numpy as np, joblib, logging

detector = joblib.load("monitoring/mmd_detector.pkl")

# Each 15-minute batch: exactly 50 rows (200 predictions/hour ÷ 4)
batch = load_latest_inference_batch(n=50)
X_current = batch[FEATURE_COLS].values.astype(np.float32)

result = detector.predict(X_current, return_p_val=True, return_distance=True)

if result["data"]["is_drift"]:
    logging.warning(f"MMDDrift ALERT: p_val={result['data']['p_val']:.4f} "
                    f"dist={result['data']['distance']:.4f}")
    send_slack_alert("drift", result["data"]["p_val"])

The detector went live on August 5, 2026. By August 8, the Slack #ml-alerts channel had received 287 drift alerts covering three days of monitoring — approximately 4.8 alerts per hour. The team opened the alert logs, compared the distribution of p-values, and noted that they appeared to be random: some windows had p=0.003, others had p=0.41, others had p=0.01, with no pattern correlating to any known production events. By August 12, the team had established an informal consensus to ignore all MMDDrift alerts. The monitoring pipeline continued running but its outputs were effectively muted.

The statistical reason for the high false positive rate is grounded in the properties of the MMD estimator at small sample sizes. The biased MMD estimator has variance that scales as:

# Statistical analysis of MMDDrift power at n=50
# Run by the retainer developer to characterize the false positive rate

import numpy as np
from alibi_detect.cd import MMDDrift

# Empirical FPR measurement: draw 500 pairs of samples from the reference distribution
# under the null hypothesis (no drift) and measure how often p < 0.05
reference_data = load_reference_data("2026-01-01", "2026-06-30")  # 5,000 rows
X_ref = reference_data[FEATURE_COLS].values.astype(np.float32)

detector = MMDDrift(X_ref, p_val=0.05, backend="pytorch")

n_trials = 500
false_positives = 0
for _ in range(n_trials):
    # Draw 50 rows from the reference distribution (null hypothesis)
    idx = np.random.choice(len(X_ref), size=50, replace=False)
    X_null = X_ref[idx]
    result = detector.predict(X_null, return_p_val=True)
    if result["data"]["is_drift"]:
        false_positives += 1

empirical_fpr = false_positives / n_trials
print(f"Empirical FPR at n=50, n_permutations=100: {empirical_fpr:.3f}")
# Output: Empirical FPR at n=50, n_permutations=100: 0.424
# Expected (nominal significance): 0.050
# Actual: 0.424 — 8.5× the nominal false positive rate

The empirical false positive rate of 42.4% was 8.5 times the nominal 5% significance level. The explanation is two-layered. First, the MMD test statistic has high variance at n=50: the finite-sample bias correction for the unbiased MMD estimator involves O(1/n²) terms, and at n=50 the variance of the estimator itself is large enough that many draws from the null hypothesis produce MMD values that look non-zero. Second, the permutation test that Alibi Detect uses to compute p-values runs n_permutations=100 permutations of the data to build the null distribution. With 100 permutations, the estimated null distribution has its own sampling variability: the standard error of the p-value estimate is approximately sqrt(p*(1-p)/100), which at p=0.05 is 0.022. Windows where the true null p-value is anywhere in [0.03, 0.07] will straddle the 0.05 threshold due to permutation variability alone. These two sources of variance compound: the MMD test statistic itself fluctuates, and the p-value estimate from the permutation test fluctuates around whatever the MMD statistic produces. The result is a p-value for 50-row windows that is far closer to a uniform[0,1] variable than the calibrated p-value the team had assumed when setting the 0.05 threshold.

On October 4, 2026, a data engineering team updated the source data warehouse ETL pipeline that feeds the inference feature store. The update was intended to standardize null handling: three columns — logins_7d, support_tickets_90d, and avg_session_duration — had previously propagated NaN for accounts with no recorded activity in the relevant window. The new ETL replaced these NaN values with 0. Approximately 22% of B2B accounts had no logins in any given 7-day window; 38% had no support tickets; 15% had no recorded sessions. The change moved large fractions of the production feature distribution to zero spikes not present in the reference distribution:

# Feature distribution shift October 4 2026 (before vs after ETL change)
# Reference (Jan–Jun 2026): NaN values were NaN, not 0

# logins_7d:
#   Reference: mean=4.2, std=3.8, null_rate=22% (NaN)
#   Post-ETL:  mean=3.3, std=3.9, null_rate=0% (NaN → 0), spike at 0: 22% of rows

# support_tickets_90d:
#   Reference: mean=1.4, std=1.9, null_rate=38% (NaN)
#   Post-ETL:  mean=0.9, std=1.9, null_rate=0% (NaN → 0), spike at 0: 38% of rows

# avg_session_duration:
#   Reference: mean=12.4, std=7.2, null_rate=15% (NaN)
#   Post-ETL:  mean=10.5, std=8.1, null_rate=0% (NaN → 0), spike at 0: 15% of rows

# Model trained on data with NaN values (imputed during training preprocessing)
# Production model receives 0 instead of NaN — preprocessing imputer sees different
# distribution than it was fitted on; churn probability estimates shift downward
# for high-churn accounts with no recorded activity (now imputed to 0 instead of NaN)

The MMDDrift detector fired an alert at the October 4 15:30 UTC monitoring window. The Slack message was identical in format to the 287 alerts that had fired in the first week of August. The on-call data scientist dismissed it without investigation. The same pattern repeated for the October 4, 5, 6, 7 windows: alerts fired, were acknowledged, and dismissed. The model’s AUC had begun degrading from October 4 because the NaN-to-zero imputation bypassed the training-time imputer and shifted the distribution of logins_7d, support_tickets_90d, and avg_session_duration in exactly the features most predictive of churn. Accounts with no logins, no tickets, and no sessions in the reference window were genuinely high-churn accounts; the model had learned this correctly. When those same accounts now appeared with 0 instead of NaN, the preprocessing imputer treated 0 as a valid low value rather than a missing value, producing systematically lower predicted churn probabilities for the accounts at highest risk.

The fix was a window accumulation buffer inserted between the inference pipeline and the drift detector. Instead of running detector.predict() on each 50-row batch, the pipeline accumulated batches into a 1,000-row buffer (20 consecutive 15-minute batches, totaling approximately 5 hours of inference) and ran the detector once per buffer fill:

# monitoring/run_detector_v2.py — fixed (October 2026)
from alibi_detect.cd import MMDDrift
import numpy as np, joblib, logging
from collections import deque

WINDOW_SIZE = 1_000   # accumulate 1,000 rows before running detector
buffer = deque(maxlen=WINDOW_SIZE)

# Re-fitted detector with increased n_permutations
X_ref = load_reference_data("2026-01-01", "2026-06-30")[FEATURE_COLS].values.astype(np.float32)
detector = MMDDrift(
    X_ref,
    p_val=0.05,
    backend="pytorch",
    n_permutations=1_000,  # increased from default 100 — reduces p-value variance
)

def process_batch(batch: np.ndarray):
    buffer.extend(batch)
    if len(buffer) < WINDOW_SIZE:
        return  # window not yet full — accumulate without running
    X_current = np.array(list(buffer), dtype=np.float32)
    result = detector.predict(X_current, return_p_val=True, return_distance=True)
    if result["data"]["is_drift"]:
        logging.warning(f"MMDDrift ALERT (n={WINDOW_SIZE}): "
                        f"p_val={result['data']['p_val']:.4f} "
                        f"dist={result['data']['distance']:.4f}")
        send_slack_alert("drift", result["data"]["p_val"])
    buffer.clear()  # reset buffer after each detection window

The retainer developer validated the corrected configuration with an empirical false positive rate measurement before deploying to production:

# Empirical FPR validation — n=1,000 window, n_permutations=1,000
n_trials = 500
false_positives = 0
for _ in range(n_trials):
    idx = np.random.choice(len(X_ref), size=1_000, replace=False)
    X_null = X_ref[idx]
    result = detector_v2.predict(X_null, return_p_val=True)
    if result["data"]["is_drift"]:
        false_positives += 1

empirical_fpr = false_positives / n_trials
print(f"Empirical FPR at n=1,000, n_permutations=1,000: {empirical_fpr:.3f}")
# Output: Empirical FPR at n=1,000, n_permutations=1,000: 0.018
# Within acceptable bounds of nominal 0.05 — confirmed calibrated detector

With the corrected configuration, the false positive rate dropped to 1.8% — well within acceptable bounds. The team re-ran the detector retroactively on the October 4–25 period using accumulated 1,000-row windows reconstructed from the inference logs. The October 4 window (first 1,000 rows after the ETL change) produced a p-value of 0.001, well below the 0.05 threshold. The NaN-to-zero drift would have been caught within 5 hours of the ETL change under the corrected configuration. The data engineering team reverted the NaN-to-zero substitution and replaced it with an explicit missing value strategy aligned with the training-time preprocessing pipeline.

Total retainer engagement: 3.0 hours (1.0h root cause: empirical FPR measurement on 500 held-out reference draws at n=50, MMD test power analysis, permutation test variance characterization, alert log review identifying 42% alert rate over August; 0.75h impact assessment: data pipeline ETL change identification via source warehouse query log audit, AUC degradation measurement from October weekly model reviews, 3-week undetected degradation scope, churn prediction accuracy impact on October cohort; 0.75h fix: 1,000-row window accumulation implementation, n_permutations=1,000 configuration, empirical FPR validation at corrected settings, monitoring pipeline deployment; 0.5h post-incident: retroactive window reconstruction on October 4–25 inference logs, NaN-to-zero ETL reversion confirmation, data engineering team handoff). Retainer hours logged to HourTab with Alibi Detect version, the MMDDrift underpowered window root cause, and the minimum window size calibration policy, so the client could audit the full 3.0-hour engagement at the shared URL without logging in.

Related retainer posts

• NannyML developer on retainer — stale CBPE calibration overestimates AUC 0.13 — 6-month retraining delay retainer work
• Evidently developer on retainer — DataDriftPreset small-window Wasserstein false positives retainer work
• Arize developer on retainer — truncated UUID prediction ID collision corrupts delayed-actual joins retainer work