Giskard developer on retainer: pandas 2.0 DatetimeTZDtype precision change silently misaligns 12% of predictions with ground truth in automated test suite — giskard 2.14.0 — 3 consecutive blocked deployments — model unchanged, test infrastructure broken — Giskard developer on monthly retainer

giskard 2.14.0 • Python 3.11 • pandas 2.0.1 • scikit-learn 1.4.2 • XGBoost 2.0.3 • PostgreSQL datetime with timezone columns • DatetimeTZDtype MultiIndex • pandas 2.0 datetime64[us] precision change • 601/5,000 rows misaligned • test_f1() reports 0.71 vs actual 0.91 • 3 blocked deployments August–September 2026 • fix: df.reset_index(drop=False) before wrap_dataset() • 3.0h retainer engagement

Northgate Analytics Ltd. runs weekly automated testing of their customer churn prediction model before each production deployment. The model — an XGBoost classifier trained on two years of subscription data — had been running in production for eight months with stable performance. In early August 2026, Northgate’s data platform team upgraded their Python environment from Python 3.10 + pandas 1.5.3 to Python 3.11 + pandas 2.0.1 as part of a routine platform modernization. The model code, training pipeline, feature engineering, and Giskard test suite were unchanged. The following Monday, the weekly deployment pipeline failed for the first time: Giskard’s test_f1() reported 0.71 for the churn model on a high-value customer segment, below the 0.80 threshold that blocked deployment. The same failure recurred the following two Mondays. Three consecutive blocked deployments. The ML team had not changed the model, the training data, or the test dataset. They spent three weeks investigating feature drift, data quality issues, and potential labeling errors in the test dataset before a Giskard retainer developer identified the actual root cause in the test infrastructure.

The Giskard test suite had been constructed as follows:

# tests/test_churn_model.py
# Northgate Analytics Ltd.
# giskard 2.14.0, Python 3.11, pandas 2.0.1

import giskard
import pandas as pd
import xgboost as xgb
import joblib

def load_test_dataset() -> pd.DataFrame:
    """Load test dataset from PostgreSQL — returns DataFrame with DatetimeTZDtype index."""
    conn = get_db_connection()
    df = pd.read_sql(
        """
        SELECT timestamp_utc, customer_segment, feature_1, feature_2,
               feature_3, feature_4, feature_5, churn
        FROM churn_test_dataset
        WHERE dataset_version = 'v4'
        ORDER BY timestamp_utc, customer_segment
        """,
        conn,
        parse_dates=["timestamp_utc"],
    )
    # Set MultiIndex for consistent ordering
    # BUG: index derived from DatetimeTZDtype — behavior changed in pandas 2.0
    df = df.set_index(["timestamp_utc", "customer_segment"])
    return df


def test_churn_model_f1():
    """Verify churn model F1 score on high-value segment."""

    # Load model and test data
    model = joblib.load("models/churn_xgb_v12.pkl")
    test_df = load_test_dataset()

    # Wrap for Giskard
    giskard_dataset = giskard.wrap_dataset(
        test_df,
        target="churn",
        cat_columns=["customer_segment"],
    )
    giskard_model = giskard.wrap_model(
        model,
        model_type="classification",
        feature_names=["feature_1", "feature_2", "feature_3", "feature_4", "feature_5"],
        classification_labels=[0, 1],
    )

    # Assert F1 score on high-value segment
    result = giskard.testing.test_f1(
        model=giskard_model,
        dataset=giskard_dataset.filter(
            lambda df: df.index.get_level_values("customer_segment") == "high_value"
        ),
        threshold=0.80,
    ).execute()

    assert result.passed, f"F1 score {result.metric:.3f} below threshold 0.80"

The bug is in how pandas 2.0 changed the internal storage of timezone-aware datetime columns returned by pd.read_sql(). In pandas 1.5.3, timezone-aware datetime columns were stored as datetime64[ns, UTC] — nanosecond precision with UTC timezone. In pandas 2.0, the same columns are stored as datetime64[us, UTC] — microsecond precision with UTC timezone — to align with Apache Arrow’s datetime representation. Northgate’s PostgreSQL database stores timestamps with microsecond precision (PostgreSQL’s native TIMESTAMP WITH TIME ZONE has microsecond precision). When a batch insertion job ran and wrote timestamps like 2026-07-15 14:23:01.000123+00:00, those timestamps were stored in PostgreSQL with the .000123 microsecond component preserved. In pandas 1.5.3, when pd.read_sql() loaded those timestamps into a datetime64[ns, UTC] column, it multiplied the microseconds to nanoseconds: 2026-07-15 14:23:01.000123000+00:00. When the DataFrame was set as a MultiIndex and Giskard performed internal alignment operations on that index during test execution, pandas 1.5.3’s internal datetime comparison path normalized sub-millisecond components in certain alignment code paths, making 2026-07-15 14:23:01.000123000+00:00 and 2026-07-15 14:23:01.000000000+00:00 hash-equivalent for index lookup purposes in those specific paths. In pandas 2.0, the datetime64[us, UTC] storage preserved microsecond precision faithfully, but the internal hashing and equality implementation for DatetimeTZDtype changed its normalization behavior. Giskard 2.14.0’s internal prediction-to-ground-truth alignment code performed a .reindex() call on the predictions Series using the stored MultiIndex. Under pandas 2.0, this .reindex() failed to match 601 out of 5,000 rows — the rows whose timestamp_utc values had non-zero microsecond components — and filled those rows with NaN rather than raising a KeyError. Giskard treated the NaN predictions as wrong predictions for F1 computation purposes. 601 wrong predictions out of 5,000 rows pushed the F1 score from 0.91 to a reported value of 0.71 for the high-value segment.

The misalignment was entirely silent. No exception, no warning, no error log. The Giskard test suite reported a numeric F1 score. The score was just wrong. The retainer developer confirmed the diagnosis by running a direct comparison outside of Giskard:

# Diagnostic: compare Giskard F1 against direct sklearn computation
import sklearn.metrics as metrics

test_df = load_test_dataset()
model = joblib.load("models/churn_xgb_v12.pkl")

# Direct sklearn computation — no index alignment involved
feature_cols = ["feature_1", "feature_2", "feature_3", "feature_4", "feature_5"]
high_value = test_df[
    test_df.index.get_level_values("customer_segment") == "high_value"
]
X_test = high_value[feature_cols]
y_true = high_value["churn"]
y_pred = model.predict(X_test)

direct_f1 = metrics.f1_score(y_true, y_pred)
print(f"Direct sklearn F1: {direct_f1:.3f}")  # Direct sklearn F1: 0.913

# Giskard test result (from CI logs): 0.71
# Discrepancy: 0.203 — 601 rows misaligned

# Check how many rows have non-zero microsecond components
timestamps = test_df.index.get_level_values("timestamp_utc")
microsecond_rows = (timestamps.microsecond != 0).sum()
print(f"Rows with non-zero microseconds: {microsecond_rows}")
# Rows with non-zero microseconds: 601

The direct sklearn F1 was 0.913 — the model was working correctly. The 601 rows with non-zero microsecond components in their timestamp_utc index level were exactly the rows Giskard misaligned. The fix was a one-line change to the dataset preparation: reset the index before wrapping, converting the DatetimeTZDtype MultiIndex to a clean RangeIndex and promoting timestamp_utc and customer_segment to regular DataFrame columns:

# Fixed version — reset index before wrapping
def load_test_dataset() -> pd.DataFrame:
    conn = get_db_connection()
    df = pd.read_sql(
        """
        SELECT timestamp_utc, customer_segment, feature_1, feature_2,
               feature_3, feature_4, feature_5, churn
        FROM churn_test_dataset
        WHERE dataset_version = 'v4'
        ORDER BY timestamp_utc, customer_segment
        """,
        conn,
        parse_dates=["timestamp_utc"],
    )
    # Keep timestamp_utc and customer_segment as regular columns
    # RangeIndex is stable across pandas versions — no datetime precision issues
    return df  # index is now default RangeIndex(start=0, stop=5000, step=1)


def test_churn_model_f1():
    test_df = load_test_dataset()
    model = joblib.load("models/churn_xgb_v12.pkl")

    giskard_dataset = giskard.wrap_dataset(
        test_df,
        target="churn",
        cat_columns=["customer_segment"],  # customer_segment is now a regular column
    )
    giskard_model = giskard.wrap_model(
        model,
        model_type="classification",
        feature_names=["feature_1", "feature_2", "feature_3", "feature_4", "feature_5"],
        classification_labels=[0, 1],
    )

    # Filter by customer_segment as a regular column
    result = giskard.testing.test_f1(
        model=giskard_model,
        dataset=giskard_dataset.filter(
            lambda df: df["customer_segment"] == "high_value"
        ),
        threshold=0.80,
    ).execute()

    assert result.passed, f"F1 score {result.metric:.3f} below threshold 0.80"

After deploying the fix, the September 9 weekly pipeline run produced a Giskard F1 of 0.913 for the high-value segment — matching the direct sklearn computation and passing the 0.80 threshold. The three-week investigation of model drift and data quality had been entirely spurious. The churn model had been performing at 0.91+ F1 throughout the entire period of blocked deployments. The Giskard test suite had been the only broken component.

The retainer developer identified a secondary risk: the filter expression in the original test used df.index.get_level_values("customer_segment") to access the MultiIndex level. After the fix, customer_segment became a regular column and the filter correctly changed to df["customer_segment"]. Any other test functions in the suite that also filtered on index levels needed to be updated. There were three additional test functions that used the same pattern. All three were updated in the same PR. The retainer developer also added a pandas version check to the test setup:

# conftest.py — warn if complex index passed to wrap_dataset
import pandas as pd
import pytest

@pytest.fixture(autouse=True)
def check_dataset_index(monkeypatch):
    """Intercept wrap_dataset() calls and warn if non-RangeIndex is used."""
    original_wrap = giskard.wrap_dataset

    def patched_wrap(df, **kwargs):
        if not isinstance(df.index, pd.RangeIndex):
            pytest.warns(UserWarning,
                match="Non-RangeIndex passed to wrap_dataset — reset_index() recommended")
        return original_wrap(df, **kwargs)

    monkeypatch.setattr(giskard, "wrap_dataset", patched_wrap)

This fixture causes any future test that passes a non-RangeIndex DataFrame to giskard.wrap_dataset() to emit a warning during the test run, making the risk visible before it causes misalignment. The warning was added rather than an error to avoid breaking any test that legitimately uses a custom index for non-alignment purposes.

The postmortem defined one new development requirement: any pandas major or minor version upgrade in a Giskard testing environment must be preceded by a Giskard test suite run on a stable model (one whose performance is known from direct sklearn evaluation), with confirmation that all Giskard test scores match the direct sklearn scores within 0.02. This validation runs in a separate CI job that gates the pandas upgrade PR. It is distinct from the regular weekly deployment pipeline.

Total retainer engagement: 3.0 hours (0.5h initial triage: compare Giskard reported F1 against direct sklearn classification_report() on the same test dataset and model; confirm 0.203 gap; rule out model change by checking git diff on model artifact hash; confirm pandas version change from 1.5.3 to 2.0.1 in the same platform upgrade PR; 0.75h root cause: inspect load_test_dataset() code and identify MultiIndex construction from datetime column; confirm 601 rows with non-zero microsecond timestamps; examine Giskard 2.14.0 source for internal alignment operations on dataset index; confirm .reindex() path and its behavior change under pandas 2.0 DatetimeTZDtype; 1.0h fix: change load_test_dataset() to return DataFrame without set_index(); update all four Giskard test filter expressions from index-level access to column access; add conftest.py warning fixture; update test documentation; 0.75h validation: run full Giskard test suite on churn model v12, confirm F1 = 0.913 for high-value segment; compare against sklearn direct computation, confirm match within 0.001; confirm all four test functions pass; document pandas 2.0 upgrade validation protocol for team runbook).

Related retainer posts

• Deepchecks developer on retainer — Deepchecks suite fails silently on nullable integer columns after pandas upgrade
• MLflow model registry developer on retainer — challenger model promoted without Giskard gate, production regression undetected for 11 days
• TruLens developer on retainer — metadata-polluted context extractor, 0.42 groundedness score for 6 months