NannyML developer on retainer: stale CBPE calibration overestimates model AUC by 0.13 after feature distribution shift — team delays retraining 6 months on false performance baseline — NannyML developer on monthly retainer

nannyml 0.10.3 • Python 3.11 • binary classification churn • 50,000 predictions/day • CBPE fitted April 2026 on Jan–Mar validation data • new acquisition channel May 2026 shifts user_age + session_count • CBPE calibration never refitted • NannyML estimated AUC 0.840 • actual AUC 0.713 • 0.127-point overestimate • 6-month retraining delay • fix: rolling 90-day CBPE refit on drift detection • 3.0h retainer engagement

In October 2026, the ML team at Cedarwood ML Solutions compared their NannyML-estimated churn model AUC against the actual AUC computed from ground-truth churn labels that had arrived 90 days after prediction — a quarterly validation they had not performed since the NannyML integration was deployed in April. NannyML’s estimated AUC for the July 2026 monitoring window was 0.840. The actual AUC from ground-truth labels for the same July predictions was 0.713. The 0.127-point gap was large enough that it had directly influenced retraining decisions: the team had reviewed the NannyML dashboard monthly, seen an estimated AUC consistently above 0.82, and had not triggered a retraining run. The NannyML retainer developer traced the discrepancy to a CBPE calibration fitted on a January–March 2026 reference dataset that had never been re-fitted after a new customer acquisition channel launched in May 2026 and shifted the feature distribution. The fix was a re-fitting pipeline. The total engagement was 3.0 hours.

Cedarwood ML Solutions uses NannyML (0.10.3) to monitor a churn prediction model that scores 50,000 customers per day. Ground-truth churn labels arrive 90 days after scoring — the model predicts which customers will churn in the next quarter, and outcomes become known only after that quarter ends. Because ground truth is delayed, the team had deployed NannyML’s Confidence-Based Performance Estimation (CBPE) to estimate model AUC on each monthly production window without waiting for labels. The CBPE estimator had been fitted once at integration time on a validation dataset drawn from the January–March 2026 training period and serialized to S3:

# monitoring/setup_cbpe.py — initial CBPE calibration (April 2026, run once)
# Cedarwood ML Solutions, nannyml 0.10.3, Python 3.11
import nannyml as nml
import joblib, boto3

# Reference: validation split from Jan–Mar 2026 training run
# 45,000 rows; features include user_age, session_count, days_since_signup,
# premium_plan, support_tickets_90d, nps_score_last, plan_tier
reference_df = load_validation_data("2026-01-01", "2026-03-31")  # 45,000 rows

estimator = nml.CBPE(
    y_pred_proba="churn_score",
    y_pred="churn_prediction",
    y_true="actual_churn",
    timestamp_column_name="scored_at",
    metrics=["roc_auc"],
    chunk_size=50_000,  # one month of scoring at 50K/day × ~30 days
    problem_type="binary_classification",
)

estimator.fit(reference_df)  # fits probability calibration on reference distribution

# Serialized April 2026, never re-fitted
joblib.dump(estimator, "/tmp/cbpe_estimator.pkl")
s3 = boto3.client("s3")
s3.upload_file("/tmp/cbpe_estimator.pkl", "cedarwood-ml", "monitoring/cbpe_estimator_2026-04-01.pkl")

The monitoring pipeline loaded this serialized estimator from S3 each month and applied it to the current production window to estimate AUC:

# monitoring/monthly_estimate.py — monthly job (unchanged April–October 2026)
import nannyml as nml
import joblib, boto3

# Always loads the April 2026 estimator — never re-fitted
s3 = boto3.client("s3")
s3.download_file("cedarwood-ml", "monitoring/cbpe_estimator_2026-04-01.pkl", "/tmp/cbpe.pkl")
estimator = joblib.load("/tmp/cbpe.pkl")

production_df = load_production_data(month_start, month_end)
results = estimator.estimate(production_df)
# Outputs: estimated_roc_auc per chunk — published to NannyML dashboard

On May 15, 2026, NannyML’s UnivariateDriftCalculator fired PSI drift alerts on user_age (PSI = 0.23, above the 0.20 threshold) and session_count (PSI = 0.19, just below threshold). The alerts were acknowledged by the team but attributed to a known new customer acquisition channel that had launched in early May — a partnership with a consumer electronics brand that brought a younger, more mobile-centric user cohort. The team noted the drift, documented it, and did not trigger a CBPE re-fitting. The decision was reasonable from the outside: the drift was expected, the source was known, and the NannyML dashboard continued to show estimated AUC above 0.82 for May, June, and July.

The problem was the relationship between the new cohort’s feature distribution and the CBPE calibration. The April 2026 CBPE calibrator had been fitted on a reference population that was predominantly users aged 35–55, with high session_count (median 42 sessions/month), relatively low support_tickets_90d (median 0.8), and a churn rate of 18%. The new acquisition cohort introduced users aged 18–28, with lower session_count (median 11 sessions/month), higher support_tickets_90d (median 2.1), and a churn rate of 34%. These users were harder for the model to classify: their behavioral signals were noisier, their sessions shorter, and their churn patterns less stable than the reference cohort’s. The model’s raw prediction scores for the new cohort’s true churners clustered closer to the decision boundary than the reference cohort’s true churners had — meaning the model was less confident, and its raw scores for positives and negatives were more overlapping.

The CBPE calibrator, however, had been fitted on a reference population where a score of 0.70 corresponded to a churn probability of approximately 0.85 — because the reference cohort’s true positives produced scores in the 0.65–0.85 range, and the calibrator learned a steep score-to-probability mapping in that region. When the same 0.70 score appeared for the new cohort’s true positive, the calibrator still mapped it to approximately 0.85 — but for the new cohort, a 0.70 score corresponded to a much lower actual churn probability because the score distributions were more compressed and overlapping. The calibrator was systematically overestimating the churn probability for the new cohort’s predictions. CBPE computes estimated AUC from these calibrated probabilities. Inflated calibrated probabilities for the more ambiguous new cohort made the CBPE estimator believe the model was more confident than it was — and since AUC measures the area under the ROC curve as a function of the score distribution, inflated probabilities on a compressed score distribution produced an overestimated AUC.

The retainer developer confirmed the diagnosis by obtaining the July 2026 ground-truth labels (which had become available in October 2026 after the 90-day delay) and computing the actual AUC directly:

# diagnosis/validate_cbpe_july.py
from sklearn.metrics import roc_auc_score
import pandas as pd

# Ground-truth labels available October 2026 for July 2026 predictions
july_df = load_scored_predictions("2026-07-01", "2026-07-31")  # 1,550,000 rows
july_actuals = load_actuals("2026-07-01", "2026-07-31")       # 90-day delayed labels

merged = july_df.merge(july_actuals, on="customer_id")
actual_auc = roc_auc_score(merged["actual_churn"], merged["churn_score"])
# actual_auc = 0.713

# NannyML estimated AUC for July 2026 (from dashboard): 0.840
# Discrepancy: 0.840 - 0.713 = 0.127 — above 0.05 threshold

# Confirm calibration staleness by comparing calibration reference to July production
ref_df = load_validation_data("2026-01-01", "2026-03-31")
july_prod_df = load_production_data("2026-07-01", "2026-07-31")

# user_age distribution shift: reference median 44 years vs July production median 29 years
# session_count distribution shift: reference median 42 vs July production median 18
# Confirms the calibration is stale relative to current production distribution

The fix replaced the single-fitted serialized CBPE estimator with a rolling re-fitting pipeline. Every 90 days — and immediately whenever NannyML’s drift calculator reported PSI above 0.20 on any monitored feature — the CBPE estimator was re-fitted on the most recent 90-day production window for which ground-truth labels were available:

# monitoring/refit_cbpe.py — runs on PSI drift alert or 90-day cadence
import nannyml as nml
import joblib, boto3
from datetime import datetime, timedelta

def refit_cbpe_on_new_reference(ground_truth_end_date: datetime):
    # Use 90-day window ending at ground_truth_end_date as new calibration reference.
    # Only possible once ground-truth labels have arrived (90-day delay).
    ref_start = ground_truth_end_date - timedelta(days=90)
    ref_df = load_labeled_production_data(ref_start, ground_truth_end_date)

    if len(ref_df) < 30_000:
        raise ValueError(f"Insufficient labeled data for CBPE calibration: {len(ref_df)} rows (need 30,000)")

    estimator = nml.CBPE(
        y_pred_proba="churn_score",
        y_pred="churn_prediction",
        y_true="actual_churn",
        timestamp_column_name="scored_at",
        metrics=["roc_auc"],
        chunk_size=50_000,
        problem_type="binary_classification",
    )
    estimator.fit(ref_df)

    # Validate before deploying: estimated AUC on ref_df vs actual AUC
    results = estimator.estimate(ref_df)
    estimated_auc = results.filter(metrics=["roc_auc"]).to_df()["estimated_roc_auc"].mean()
    actual_auc = roc_auc_score(ref_df["actual_churn"], ref_df["churn_score"])

    if abs(estimated_auc - actual_auc) > 0.05:
        raise ValueError(
            f"CBPE calibration quality check failed: estimated {estimated_auc:.3f} vs actual {actual_auc:.3f}"
        )

    timestamp = ground_truth_end_date.strftime("%Y-%m-%d")
    joblib.dump(estimator, f"/tmp/cbpe_{timestamp}.pkl")
    s3 = boto3.client("s3")
    s3.upload_file(f"/tmp/cbpe_{timestamp}.pkl", "cedarwood-ml", f"monitoring/cbpe_estimator_{timestamp}.pkl")
    # Also write a 'latest' pointer so monthly_estimate.py loads the newest estimator
    s3.put_object(Bucket="cedarwood-ml", Key="monitoring/cbpe_latest.txt",
                  Body=f"monitoring/cbpe_estimator_{timestamp}.pkl".encode())

The first re-fitted CBPE estimator used the labeled April–June 2026 production window as the calibration reference — the earliest post-drift window for which 90-day ground-truth labels were available. The re-fitted estimator’s estimated AUC on the July 2026 production window was 0.719, compared to the actual AUC of 0.713 — a discrepancy of 0.006 AUC, well within the acceptable 0.05 threshold. The team initiated a model retraining run in October 2026 using data from the post-acquisition-channel period. The retrained model, deployed November 2026, scored 0.768 AUC on the October ground-truth window — a 7.8-point improvement over the degraded production model.

Total retainer engagement: 3.0 hours (1.0h root cause: ground-truth vs NannyML estimate comparison, CBPE calibration staleness hypothesis, May drift alert timeline correlation with acquisition channel launch, calibration reference vs July production distribution comparison; 0.5h impact assessment: 6-month retraining delay quantification, customer impact from degraded model in production May–October 2026, retraining timeline recommendation; 1.0h fix: rolling 90-day CBPE re-fitting pipeline, validation gate requiring estimated vs actual AUC within 0.05, GitHub Actions cron for 90-day cadence re-fitting, drift-triggered re-fitting webhook from NannyML PSI alert; 0.5h post-incident: first re-fit on April–June 2026 labeled data, estimated AUC validation, deployment to production monitoring pipeline, team documentation of CBPE calibration freshness requirement). Retainer billing was logged to HourTab with the NannyML version, the CBPE calibration staleness mechanism, and the complete rolling re-fit architecture, so the client could audit the full 3.0-hour work log at the shared URL without logging in.

Related retainer posts

Track retainer hours without a client portal

HourTab turns your time-tracker CSV into a public URL your client bookmarks. No login. No portal. CSV in, URL out.

See pricing →