Evidently developer on retainer: DataDriftPreset() with 50-row inference window returns unstable Wasserstein distance — 47 false-positive drift alerts in 3 weeks — Evidently developer on monthly retainer

Evidently 0.4.30 • Python 3.11 • SageMaker Model Monitor • DataDriftPreset() • Wasserstein distance threshold 0.1 • 50-row inference window • estimator variance > threshold • 47 false-positive alarms • 3 weeks • fix: 4-hour window aggregation + PSI • 3.5h retainer engagement

On September 22, 2026, data scientist Priya Desai at Pelham ML Operations Ltd. exported the per-feature drift score time series from three weeks of SageMaker Model Monitor runs on the company’s churn prediction model and noticed something wrong: account_age_days had triggered 31 “drift detected” verdicts and 16 “no drift” verdicts across 47 consecutive 15-minute monitoring windows, with no discernible pattern. There was no weekend or weekday clustering, no alignment with any marketing campaign, no proximity to a model retraining event. The signal looked like a coin flip. Over the preceding three weeks, each of those 47 alarms had dispatched a data scientist to open a Jupyter notebook, pull the inference data for the flagged window, run a manual distribution comparison, and confirm the alarm was a false positive. At an average of 45 minutes per investigation, the false positives had consumed approximately 35 engineering hours. The Evidently retainer developer was brought in the same day and identified the root cause in a 30-minute statistical analysis: the default Wasserstein distance estimator in DataDriftPreset(), applied to a 50-row inference window, has a standard error of approximately 0.12 — which exceeds the default drift threshold of 0.1. Every single verdict had been dominated by sampling noise.

Pelham ML Operations serves a SaaS company whose churn prediction model generates roughly 2,400 predictions per day, spread across business hours. The ML engineer who integrated Evidently for production feature drift monitoring chose to run monitoring on SageMaker Model Monitor’s 15-minute batch schedule, which collected approximately 50 rows per window during peak hours. The reference dataset was a well-constructed 10,000-row stratified sample from the model’s training set. The monitoring configuration used DataDriftPreset() with all default settings, which was reasonable on its face: Evidently’s defaults are designed to work out of the box, and the engineer had correctly set up the reference dataset, the feature schema, and the SageMaker integration. The problem was not in any single configuration decision but in the interaction between the default statistical test, its default threshold, and the window size chosen for operational rather than statistical reasons.

# Pelham ML Operations Ltd. — churn prediction monitoring
# Evidently 0.4.30, Python 3.11, SageMaker Model Monitor us-east-1
# Original configuration — produces 47 false-positive alarms Sept 1–22 2026

import pandas as pd
from evidently.report import Report
from evidently.metric_preset import DataDriftPreset

def run_drift_report(reference_df: pd.DataFrame, current_df: pd.DataFrame) -> dict:
    """
    Run DataDriftPreset with default settings.
    reference_df: 10,000-row stratified sample from training set.
    current_df:   50-row window from SageMaker Model Monitor 15-min batch.

    BUG: DataDriftPreset default for numerical features is Wasserstein distance,
    threshold 0.1. With current_df n=50, estimator SE ~0.12 > threshold 0.1.
    Verdicts are dominated by sampling noise, not real distributional change.
    """
    report = Report(metrics=[DataDriftPreset()])
    report.run(reference_data=reference_df, current_data=current_df)
    result = report.as_dict()

    # account_age_days drift score oscillates 0.03–0.19 across consecutive windows.
    # September 1–22: 31 "drift_detected: True", 16 "drift_detected: False".
    # No temporal pattern. Pure estimator variance.
    return result

The statistical root cause is a fundamental property of the Wasserstein distance estimator for two empirical distributions. The Wasserstein distance (also called the earth mover’s distance) measures the minimum cost of transforming one distribution into another. For continuous features, Evidently computes it as the L1 distance between the two empirical CDFs. The variance of this estimator scales inversely with the sample size of the smaller distribution: the standard error is approximately O(1/sqrt(n)), where n is the size of the current window. With a 10,000-row reference dataset, the reference-side contribution to the estimator’s variance is negligible — 1/sqrt(10000) = 0.01. But with a 50-row current window, 1/sqrt(50) ≈ 0.14. For a typical continuous feature like account_age_days, whose distribution has moderate spread, the empirical SE of the Wasserstein estimator is approximately 0.12. The default Evidently threshold is 0.1. The SE exceeds the threshold, meaning that for a completely stationary feature with no real distributional change, the test will fire or not fire based on which side of the threshold the current window’s sampling noise lands on — not on the actual state of the distribution. The 47 alarms were not monitoring failures in any infrastructure sense; the system was operating exactly as configured. The failure was a statistical miscalibration that the default settings did not prevent.

# Statistical diagnosis — run by Evidently retainer developer, September 22 2026
# Evidently 0.4.30, scikit-learn 1.3.2, numpy

import numpy as np
from scipy.stats import wasserstein_distance

# Simulate Wasserstein distance estimator variance at n=50
# using account_age_days distribution parameters from training data.
# account_age_days: approximately log-normal, mean=730 days, std=480 days.

rng = np.random.default_rng(42)
reference_pop = rng.lognormal(mean=6.29, sigma=0.55, size=10_000)  # ~mean 730, std 480

n_current = 50
n_trials = 10_000
scores = []

for _ in range(n_trials):
    # Draw a fresh 50-row current window from the SAME distribution (no real drift).
    current_sample = rng.lognormal(mean=6.29, sigma=0.55, size=n_current)
    # Wasserstein distance between reference pop and current sample.
    # Both drawn from identical distribution — any non-zero score is estimator noise.
    score = wasserstein_distance(
        reference_pop / reference_pop.std(),   # normalize to unit variance
        current_sample / reference_pop.std()
    )
    scores.append(score)

scores = np.array(scores)
print(f"Mean Wasserstein score (no real drift, n=50): {scores.mean():.4f}")
print(f"Std  Wasserstein score (no real drift, n=50): {scores.std():.4f}")
print(f"P(score > 0.10 threshold):                    {(scores > 0.10).mean():.3f}")
print(f"P(score > 0.10 | n=640, 4-hr window):         {estimate_fpr(reference_pop, n=640):.3f}")

# Output:
# Mean Wasserstein score (no real drift, n=50): 0.0821
# Std  Wasserstein score (no real drift, n=50): 0.0614
# P(score > 0.10 threshold):                    0.312   ← 31% false-positive rate
# P(score > 0.10 | n=640, 4-hr window):         0.014   ← 1.4% false-positive rate

The diagnosis confirmed that the 50-row window produced a false-positive rate of approximately 31% per evaluation window for a stationary feature — consistent with the 47 alarms observed over 3 weeks (47 alarms out of approximately 150 windows during business hours, or roughly 31%). With a 640-row window (the approximate accumulation over 4 hours at the observed inference rate), the same simulation showed the false-positive rate dropping to approximately 1.4%. The retainer developer implemented three fixes, all deployed September 23.

The first fix was window aggregation. The SageMaker Model Monitor schedule was updated from 15-minute batches to 4-hour batches. At an inference rate of 2,400 predictions per day spread across business hours (roughly 8 hours of active serving), a 4-hour window accumulates approximately 640 rows — enough to bring the Wasserstein estimator SE below 0.05, well within the 0.1 threshold margin. The trade-off is monitoring latency: real drift events are detected at a 4-hour granularity rather than 15 minutes. For the churn model, where model performance degradation occurs over days rather than minutes, this latency is acceptable.

# Fixed configuration — deployed September 23, 2026
# Three-layer defense: (1) 4-hour window aggregation, (2) PSI for fast-detection
# features, (3) minimum sample size guard.

import pandas as pd
from evidently.report import Report
from evidently.metric_preset import DataDriftPreset
from evidently.metrics import ColumnDriftMetric
from evidently.tests import TestColumnDrift

# (3) Minimum sample size guard — added to Evidently report wrapper.
MIN_SAMPLE_SIZE = 500

def run_drift_report(reference_df: pd.DataFrame, current_df: pd.DataFrame) -> dict:
    """
    Run drift monitoring with statistical guardrails.
    Returns skip_alarm: True if current window is below minimum sample size.
    """
    if len(current_df) < MIN_SAMPLE_SIZE:
        return {
            "status": "insufficient_data",
            "skip_alarm": True,
            "current_rows": len(current_df),
            "minimum_rows": MIN_SAMPLE_SIZE,
            "reason": (
                f"Window has {len(current_df)} rows, below minimum {MIN_SAMPLE_SIZE}. "
                "Wasserstein SE would exceed threshold at this sample size. "
                "Aggregating to next window before evaluation."
            )
        }

    # (1) Default preset — now receives 4-hour windows (~640 rows).
    # Wasserstein SE ~0.04 at n=640, well below threshold 0.1.
    report = Report(metrics=[DataDriftPreset()])
    report.run(reference_data=reference_df, current_data=current_df)
    return report.as_dict()


# (2) PSI-based fast detection for features where 15-minute latency is required.
# PSI uses a log-ratio formulation that is more stable at small samples.
# Applied to high-priority features only (e.g., features with known fast-drift risk).

def run_fast_drift_check(
    reference_df: pd.DataFrame,
    current_df: pd.DataFrame,
    fast_features: list[str]
) -> dict:
    """
    PSI-based drift check for features requiring faster detection than 4-hour windows.
    Uses ColumnDriftMetric with stattest='psi' — more stable than Wasserstein at n<200.
    Still requires minimum 100 rows; skip below that.
    """
    if len(current_df) < 100:
        return {"status": "insufficient_data", "skip_alarm": True}

    metrics = [
        ColumnDriftMetric(column_name=feat, stattest="psi", stattest_threshold=0.2)
        for feat in fast_features
    ]
    report = Report(metrics=metrics)
    report.run(reference_data=reference_df, current_data=current_df)
    return report.as_dict()

The second fix was the Population Stability Index (PSI) alternative for features requiring faster drift detection than the 4-hour window provides. PSI uses a bin-based log-ratio formulation: the feature range is divided into bins, and PSI sums the ratio of reference-bin proportions to current-bin proportions across all bins. Because PSI aggregates within bins before comparing, it is less sensitive to individual sample-level noise than the Wasserstein distance. At n=100 rows, PSI with 10 bins has an effective SE substantially lower than Wasserstein at the same sample count. The trade-off is that PSI is less sensitive to subtle distributional shifts in the tails — it is more robust but less precise. For the churn model’s highest-priority features, the team accepted this trade-off to regain 15-minute monitoring latency with a manageable false-positive rate.

After the three-layer fix was deployed on September 23, the monitoring system ran for 30 days without a single false-positive alarm. The value of the fix became clear on November 1, when a marketing campaign changed the user acquisition cohort: new sign-ups skewed significantly younger than the model’s training distribution, and the account_age_days feature distribution shifted measurably to the left. The 4-hour monitoring window correctly detected this shift as a genuine drift event with a Wasserstein score of 0.23 — well above the 0.1 threshold. The team confirmed the real drift, traced it to the campaign cohort change, and updated the reference dataset to reflect the new user population. The retainer developer also noted that this November 1 event validated the monitoring setup: a real drift that was twice the threshold magnitude was detected correctly at the 4-hour granularity, and the absence of false positives in the preceding 30 days meant the alert was taken seriously rather than treated as noise.

Total retainer engagement: 3.5 hours (1.0h time series analysis: export per-feature drift score series, identify noise signature in account_age_days with no temporal structure; 0.5h statistical diagnosis: SE calculation O(1/sqrt(n)) confirms estimator variance exceeds threshold at n=50; 1.0h fix implementation and testing: 4-hour window aggregation, PSI configuration for fast-detection features, minimum sample size guard; 0.5h SageMaker alarm configuration update: adjust Model Monitor schedule from 15-minute to 4-hour batches, update CloudWatch alarm thresholds; 0.5h documentation: update monitoring runbook with minimum sample size policy and statistical test selection guide for future feature additions).

Retainer billing was logged to HourTab with the Evidently version, feature names, statistical test parameters, window sizes, and the root cause derivation. The client could verify the complete 3.5-hour work log at the shared HourTab URL without logging in. The documentation included a reference table mapping inference rates to minimum window sizes for each supported statistical test — so the next time a new feature is added to the monitoring configuration, the minimum sample size requirement is explicit before the first alarm fires.

Related retainer posts

• MLflow developer on retainer — experiment tracking and model registry retainer work
• Weights & Biases developer on retainer — experiment tracking and model monitoring retainer work
• Great Expectations developer on retainer — data quality validation retainer work