Deepchecks developer on retainer: customer_segment encoded as int64 bypasses CategoryMismatchTrainTest check — new enterprise segment code 4 enters production undetected for 90 days — 765,000 mis-scored predictions — Deepchecks developer on monthly retainer

deepchecks 0.18.0 • Python 3.11 • scikit-learn 1.3.2 • tabular binary classification churn • 20,000 predictions/day • customer_segment ETL-encoded as int64 in Parquet • Dataset() without cat_features • Deepchecks infers numeric • CategoryMismatchTrainTest never runs • segment code 4 enters production July 2026 • model predicts 11% churn • actual 43% churn • 765,000 mis-scored predictions • fix: cat_features parameter • 2.5h retainer engagement

In October 2026, the customer success team at Meridian Predictive Analytics Ltd. raised an escalation to the data science department: the enterprise expansion segment, introduced as part of a new channel partnership program in July, was churning at a rate three times higher than the CRM dashboard’s churn risk scores indicated. The model was scoring enterprise expansion accounts as low-risk when a large fraction of them were actively cancelling. The Deepchecks retainer developer was engaged to trace the source of the model’s blind spot. The root cause was a feature type inference error introduced at the ETL layer in 2025 and never caught by the Deepchecks validation suite: the customer_segment column was stored as int64 in Parquet, and the Dataset() constructor was called without the cat_features parameter. Every Deepchecks check that would have caught the new segment was bypassed. The total engagement was 2.5 hours.

Meridian Predictive Analytics runs a churn prediction model on 20,000 enterprise software customers per day. The model is a GradientBoostingClassifier trained on 14 features, one of which is customer_segment — the customer’s commercial segment at the time of scoring. The feature was originally stored as a string ('smb', 'mid_market', 'enterprise', 'startup') in the source Postgres database. In early 2025, an ETL migration to Parquet-backed Iceberg tables introduced a transformation step that converted customer_segment to integer codes using pd.Categorical(df['customer_segment']).codes. The encoded values were smb=0, mid_market=1, enterprise=2, startup=3. The Parquet schema stored the column as int64. The intent was to simplify downstream consumers. The side effect was invisible to the data science team because the model’s LabelEncoder handled the mapping, and model performance on the existing segments was unaffected.

The Deepchecks integration was built shortly after the Parquet migration. The data scientist who built it followed the standard pattern but omitted the cat_features parameter:

# validation/run_suite.py — Meridian Predictive Analytics, deepchecks 0.18.0
# Executed in CI after each training run
import deepchecks.tabular as dc
import pandas as pd

train_df = pd.read_parquet("data/train_2026.parquet")
test_df  = pd.read_parquet("data/test_2026.parquet")

# customer_segment is int64 (Parquet) — not declared as categorical
# Deepchecks infers dtype: int64 → continuous numeric feature
train_ds = dc.Dataset(train_df, label="churned", index_col="customer_id")
test_ds  = dc.Dataset(test_df,  label="churned", index_col="customer_id")

suite = dc.suites.train_test_validation()
result = suite.run(train_dataset=train_ds, test_dataset=test_ds)
result.save_as_html("reports/validation_latest.html")

Because cat_features was not specified, Deepchecks inferred the feature types from pandas dtype. Every int64 column was classified as a continuous numeric feature. The customer_segment column, with values in {0, 1, 2, 3}, was treated as a continuous numeric variable with a range of 0–3. The train_test_validation() suite ran TrainTestFeatureDrift (KolmogorovSmirnov test on numeric distributions), TrainTestPerformance, and DatasetsSizeComparison. It did not run CategoryMismatchTrainTest, because that check only applies to columns that Deepchecks has classified as categorical.

On the training data (January–June 2026), the customer_segment integer distribution had a mean of 1.31 and was spread across all four codes. The test split had a mean of 1.34. The KS statistic was 0.07, p-value 0.31: stable, no alert. The CI pipeline passed every validation run from early 2025 through July 2026. The Deepchecks HTML reports were generated and archived, and the green all-pass status built confidence that the validation was comprehensive. The CategoryMismatchTrainTest check that would have validated segment distribution was never in the report at all — not failing, not passing, simply not present, because the column was not recognized as categorical.

In July 2026, Meridian launched an enterprise expansion partnership with three SaaS platform vendors. New accounts onboarded through the partnership received a distinct commercial classification: enterprise_expansion, encoded as segment code 4. These accounts were structured differently from the existing four segments: they signed on promotional pricing terms with a 90-day evaluation window, a significantly higher early-churn rate (43% in the first 90 days, compared to 8–12% for standard enterprise accounts), and a service footprint typical of newly onboarded accounts rather than mature retainers. By August 2026, approximately 8,500 enterprise expansion accounts were being scored daily — 18% of the total daily scoring volume.

The model had been trained on segments 0–3. It had never seen segment code 4. Because customer_segment was stored as a raw integer feature, the model treated code 4 as a continuous value numerically adjacent to code 3 (enterprise). Enterprise accounts had the lowest churn rate of all segments — approximately 8% per 90-day cycle — because they were large, long-tenure contracts with high switching costs. The model’s learned association between customer_segment values near 3 and low churn carried over to code 4. Enterprise expansion accounts, most of which had other low-churn indicators on their surface (large contract_value, high initial logins, low support_tickets in the first 30 days of onboarding), received predicted churn probabilities concentrated in the 0.09–0.14 range.

The Deepchecks validation pipeline did not catch the problem. After the July model refresh training run, the CI suite ran against the new training data:

# CI validation output — July 2026 training run (deepchecks 0.18.0)
# customer_segment in training: {0, 1, 2, 3}
# customer_segment in test split: {0, 1, 2, 3}
# (enterprise_expansion segment code 4 not yet in training data — first live accounts July 15)

TrainTestFeatureDrift (customer_segment as NUMERIC):
  KS statistic: 0.07 | p-value: 0.31 | Status: PASSED (threshold: p < 0.05)
  PSI: 0.03 | Status: PASSED (threshold: PSI > 0.20)

# CategoryMismatchTrainTest: NOT RUN (customer_segment not in cat_features)

Suite result: ALL CHECKS PASSED

When the production scoring pipeline began processing enterprise expansion accounts from July 15 onwards, the customer_segment=4 values entered production data without triggering any Deepchecks alert. The numeric KS drift statistic on customer_segment measured the distribution of {0, 1, 2, 3, 4} values in production batches against the reference {0, 1, 2, 3} distribution. As the enterprise expansion program scaled, the production mean shifted from 1.31 to approximately 1.80. By mid-August, the KS statistic had climbed to 0.18 with a p-value of 0.09 — below the 0.05 significance threshold. The Deepchecks monitoring pipeline logged this as a non-alerting marginal drift event. No categorical out-of-distribution alert fired because, in the absence of cat_features, no such check was running.

The retainer developer confirmed the root cause by running a corrected Deepchecks validation with explicit categorical feature declarations:

# diagnosis/validate_with_catfeatures.py
import deepchecks.tabular as dc
import pandas as pd

# October 2026 — diagnose the July–October gap
train_df = pd.read_parquet("data/train_2026_july.parquet")   # segments 0–3
prod_df  = pd.read_parquet("data/production_2026_aug.parquet")  # includes segment 4

# Correct: declare customer_segment as categorical
train_ds = dc.Dataset(
    train_df, label="churned", index_col="customer_id",
    cat_features=["customer_segment", "contract_type", "billing_cycle"]
)
prod_ds = dc.Dataset(
    prod_df, label=None, index_col="customer_id",
    cat_features=["customer_segment", "contract_type", "billing_cycle"]
)

check = dc.checks.CategoryMismatchTrainTest()
result = check.run(train_ds, prod_ds)
result.show()

# Output:
# CategoryMismatchTrainTest: FAILED
#   customer_segment: NEW CATEGORIES IN PRODUCTION NOT SEEN IN TRAINING
#     Value 4: found in 18.3% of production rows (8,521 of 46,500)
#     New category fraction: 0.183 — ABOVE threshold 0.00 (any new category = FAIL)

The check would have failed immediately on the first production batch containing segment 4 accounts had cat_features been declared correctly. The fix was a two-line change to the Dataset() constructor, deployed retroactively as a CI gate update and integrated into the production monitoring pipeline:

# validation/run_suite.py — corrected (October 2026)
import deepchecks.tabular as dc
import pandas as pd

# Feature type registry — maintained alongside model training code
CAT_FEATURES = [
    "customer_segment",    # int64 in Parquet but semantically categorical
    "contract_type",       # str: 'monthly', 'annual', 'multi_year'
    "billing_cycle",       # str: 'prepaid', 'postpaid'
    "industry_vertical",   # str: 'saas', 'fintech', 'healthcare', 'logistics', 'other'
    "support_tier",        # str: 'standard', 'premium', 'enterprise'
]

train_ds = dc.Dataset(
    train_df, label="churned", index_col="customer_id",
    cat_features=CAT_FEATURES
)
test_ds = dc.Dataset(
    test_df, label="churned", index_col="customer_id",
    cat_features=CAT_FEATURES
)

suite = dc.suites.train_test_validation()
result = suite.run(train_dataset=train_ds, test_dataset=test_ds)

# Now runs: CategoryMismatchTrainTest, WeakSegmentsPerformance (by category),
# FeatureLabelCorrelation (treating customer_segment as categorical),
# TrainTestFeatureDrift (using chi-squared test instead of KS on categoricals)
result.save_as_html("reports/validation_latest.html")

A secondary fix added a CI assertion that compared the Deepchecks-inferred feature types against the feature type registry before the suite run, catching any future drift between the schema registry and the Dataset inferred types. The CategoryMismatchTrainTest check was configured with a condition of add_condition_new_category_ratio_less_or_equal(max_ratio=0.0) — any new production category value not seen in training caused the check to fail, blocking model promotion until the new segment was added to training data.

For enterprise expansion accounts already scored during the July–October gap, the team used the corrected model (retrained October 2026 on a dataset that included labeled enterprise expansion accounts from the first cohort) to re-score the 90-day backlog of 765,000 predictions. Accounts with re-scored churn probability above 0.30 were flagged for proactive CSM outreach. The retrained model scored enterprise expansion accounts at a mean churn probability of 0.39, compared to 0.11 from the original model — consistent with the observed 43% churn rate.

Total retainer engagement: 2.5 hours (0.75h root cause: Deepchecks report comparison between training-time validation run and the corrected cat_features validation run, feature type inference investigation via Dataset._feature_columns_info(), ETL int64 encoding confirmation against source Postgres schema, absence of CategoryMismatchTrainTest in all archived CI reports; 0.5h impact assessment: 90-day mis-scoring scope calculation at 8,500 accounts per day, enterprise expansion churn rate measurement against model predictions, CSM escalation timeline correlation with July program launch; 0.75h fix: cat_features parameter addition and feature type registry creation, CategoryMismatchTrainTest condition configuration with max_ratio=0.0, CI assertion comparing inferred types to registry, production monitoring pipeline update to include categorical drift checks; 0.5h post-incident: first corrected validation run confirming CategoryMismatchTrainTest fires for code 4, retrained model promotion, re-scoring pipeline for the 90-day gap). Retainer hours logged to HourTab with the Deepchecks version, the cat_features omission root cause, and the feature type registry policy, so the client could audit the full 2.5-hour engagement at the shared URL without logging in.

Related retainer posts

• Great Expectations developer on retainer — expectation suite column type mismatch silent validation pass retainer work
• Pandera developer on retainer — coerce=True strips leading zeros from zero-padded IDs — 28K records misjoined retainer work
• Evidently developer on retainer — DataDriftPreset small-window Wasserstein false positives retainer work