Featuretools developer on retainer: signup_date stored as object dtype skips DatetimeDTFeaturizer — zero YEAR/MONTH/DIFF primitives in 847-feature DFS output — AUC 0.68 vs temporal baseline 0.81 — 3 weeks — Featuretools developer on monthly retainer

featuretools 1.28.x • Python 3.11 • pandas 2.1.x • ft.dfs(entityset=es, target_dataframe_name='customers', max_depth=2) • EntitySet: customers, transactions, products tables • customers.signup_date: dtype object (string '2022-01-15') • Woodwork infers Unknown logical type instead of Datetime • DatetimeDTFeaturizer skipped: no DAY, MONTH, YEAR, WEEKDAY, or DIFF features created • developer expected 847 DFS features to include YEAR(signup_date), MONTH(signup_date), days_since_signup • none present • AUC=0.68 • manual LightGBM with tenure features: AUC=0.81 • developer checks featuretools version and feature list length for 3 weeks • fix: pd.to_datetime(es['customers']['signup_date']) before DFS • 847→912 features (65 new temporal features) • AUC 0.68→0.80 • 1.5h retainer engagement

Northfield Featuretools DFS Ltd. used featuretools 1.28.x to build a customer churn model on a dataset with three related tables: 12,000 customers, 340,000 transactions, and 850 products. The team constructed an EntitySet, defined two relationships (customers→transactions and products→transactions), and ran ft.dfs(entityset=es, target_dataframe_name='customers', max_depth=2). DFS produced 847 features. The best model trained on this feature matrix achieved AUC=0.68. A manual LightGBM trained with hand-crafted tenure features (days since signup, signup month, signup year) on the same data achieved AUC=0.81. The developer spent three weeks checking featuretools version, adjusting max_depth from 1 to 3, and counting feature list lengths before engaging a Featuretools retainer developer.

The EntitySet construction and DFS call at the time of the engagement:

# Northfield Featuretools DFS Ltd. — customer churn prediction
# featuretools 1.28.x, Python 3.11, pandas 2.1.x

import pandas as pd
import numpy as np
import featuretools as ft
from sklearn.model_selection import train_test_split
from sklearn.metrics import roc_auc_score
import lightgbm as lgb

# Load the three source tables
customers_df = pd.read_csv('northfield_customers_12k.csv')
# columns: customer_id, signup_date, country, plan_tier, age_band, churned
# signup_date dtype: object — values like '2022-01-15', '2021-11-03', ...
# ← Problem: pandas reads CSV dates as strings; dtype is object, not datetime64

transactions_df = pd.read_csv('northfield_transactions_340k.csv')
# columns: transaction_id, customer_id, product_id, amount, txn_date, channel

products_df = pd.read_csv('northfield_products_850.csv')
# columns: product_id, category, unit_price, supplier_id

print(f"customers.signup_date dtype: {customers_df['signup_date'].dtype}")
# customers.signup_date dtype: object
# ← Featuretools sees object dtype → Woodwork infers Unknown logical type
# ← DatetimeDTFeaturizer only fires for Datetime logical type → skipped

# Build the EntitySet — signup_date enters as object column
es = ft.EntitySet(id='northfield_churn')

es = es.add_dataframe(
    dataframe_name='customers',
    dataframe=customers_df,
    index='customer_id',
    # signup_date is NOT cast here — it enters the EntitySet as object dtype
)

es = es.add_dataframe(
    dataframe_name='transactions',
    dataframe=transactions_df,
    index='transaction_id',
    time_index='txn_date',
)

es = es.add_dataframe(
    dataframe_name='products',
    dataframe=products_df,
    index='product_id',
)

es = es.add_relationship('customers', 'customer_id', 'transactions', 'customer_id')
es = es.add_relationship('products', 'product_id', 'transactions', 'product_id')

# Run DFS — max_depth=2 to capture cross-table aggregations
features, feature_defs = ft.dfs(
    entityset=es,
    target_dataframe_name='customers',
    max_depth=2,
)

print(f"Total DFS features: {len(feature_defs)}")
# Total DFS features: 847
# Developer sees 847 and assumes temporal features are covered
# ← No YEAR(signup_date), MONTH(signup_date), DIFF(signup_date) features present

# Check for temporal feature names
temporal = [f.get_name() for f in feature_defs
            if any(p in f.get_name() for p in ['YEAR', 'MONTH', 'DAY', 'WEEKDAY', 'DIFF'])]
print(f"Temporal features found: {len(temporal)}")
# Temporal features found: 0
# ← DatetimeDTFeaturizer was never applied to signup_date

# Train churn model on the 847-feature DFS matrix
y = customers_df.set_index('customer_id')['churned'].loc[features.index]
X_train, X_test, y_train, y_test = train_test_split(
    features, y, test_size=0.2, random_state=42, stratify=y
)

model = lgb.LGBMClassifier(n_estimators=300, random_state=42)
model.fit(X_train, y_train)
auc = roc_auc_score(y_test, model.predict_proba(X_test)[:, 1])
print(f"DFS model AUC: {auc:.4f}")
# DFS model AUC: 0.6803

# Manual LightGBM with hand-crafted temporal features (the gap reference):
customers_df['signup_year']  = pd.to_datetime(customers_df['signup_date']).dt.year
customers_df['signup_month'] = pd.to_datetime(customers_df['signup_date']).dt.month
customers_df['days_since_signup'] = (
    pd.Timestamp('2024-01-01') - pd.to_datetime(customers_df['signup_date'])
).dt.days

# AUC with manual temporal features:  0.8104
# Gap: 0.13 AUC points — entirely due to missing temporal DFS features

Why object dtype silently bypasses DatetimeDTFeaturizer in Featuretools DFS

Featuretools uses Woodwork to manage column metadata inside an EntitySet. Before DFS runs, Woodwork inspects each column’s pandas dtype and assigns a logical type — a semantic classification that controls which primitives are eligible for that column. A column with pandas dtype datetime64[ns] receives the Woodwork logical type Datetime. A column with dtype object (Python string) receives the logical type Unknown or Categorical depending on cardinality.

DatetimeDTFeaturizer is a transform primitive with the input requirement ColumnSchema(logical_type=Datetime). Because signup_date enters the EntitySet as dtype object and receives logical type Unknown, the primitive dispatcher never matches DatetimeDTFeaturizer against the column. No YEAR, MONTH, DAY, WEEKDAY, HOUR, or DIFF features are generated. For string columns, Featuretools applies only the primitives compatible with Unknown or Categorical logical types: COUNT, NUM_UNIQUE, and MODE at aggregation depth, and no transform primitives because transforms require more specific types.

The failure is entirely silent. Featuretools does not emit a warning that a column named signup_date was treated as a string. The feature count (847) looks substantial. Unless a developer explicitly checks feature names for temporal primitive prefixes, the absence of YEAR/MONTH/DIFF features is undetectable from the DFS output summary. The AUC gap of 0.13 points (0.68 vs 0.81) reflects the full predictive value of customer tenure, signup seasonality, and time-based cohort effects that were left out of the feature matrix.

# Diagnostic: inspect Woodwork logical types and find missing temporal primitives
# featuretools 1.28.x — identify DatetimeDTFeaturizer skips

import featuretools as ft
import pandas as pd

# Step 1: check Woodwork logical type for each column in the EntitySet
print("=== customers table — Woodwork column metadata ===")
for col_name, col_meta in es['customers'].ww.columns.items():
    print(f"  {col_name:25s}  dtype={str(es['customers'][col_name].dtype):15s}  "
          f"logical_type={type(col_meta.logical_type).__name__}")

# Output:
#   customer_id               dtype=int64            logical_type=Integer
#   signup_date               dtype=object           logical_type=Unknown
#   ← DatetimeDTFeaturizer requires Datetime — Unknown does not qualify
#   country                   dtype=object           logical_type=Categorical
#   plan_tier                 dtype=object           logical_type=Categorical
#   age_band                  dtype=object           logical_type=Categorical
#   churned                   dtype=int64            logical_type=BooleanNullable

# Step 2: list all available DatetimeDTFeaturizer primitives and their input types
all_primitives = ft.list_primitives()
datetime_prims = all_primitives[all_primitives['valid_inputs'].str.contains('Datetime', na=False)]
print(f"\nPrimitives requiring Datetime logical type: {len(datetime_prims)}")
print(datetime_prims[['name', 'type', 'valid_inputs']].to_string(index=False))

# Output (abbreviated):
#   name                 type        valid_inputs
#   DatetimeDTFeaturizer transform   [ColumnSchema(logical_type=Datetime)]
#   ← This primitive fires ONLY for Datetime logical type columns

# Step 3: audit the full DFS feature list for temporal primitive coverage
feature_names = [f.get_name() for f in feature_defs]

print(f"\nTotal features: {len(feature_names)}")

for prefix in ['YEAR', 'MONTH', 'DAY', 'WEEKDAY', 'HOUR', 'DIFF']:
    matches = [name for name in feature_names if prefix in name]
    print(f"  {prefix} features: {len(matches)}")
    if matches:
        for m in matches[:3]:
            print(f"    {m}")

# Output:
#   Total features: 847
#   YEAR features: 0     ← DatetimeDTFeaturizer was skipped
#   MONTH features: 0    ← DatetimeDTFeaturizer was skipped
#   DAY features: 0      ← DatetimeDTFeaturizer was skipped
#   WEEKDAY features: 0  ← DatetimeDTFeaturizer was skipped
#   HOUR features: 0     ← DatetimeDTFeaturizer was skipped
#   DIFF features: 0     ← DatetimeDTFeaturizer was skipped

# Step 4: check what primitives DID fire on signup_date
signup_features = [name for name in feature_names if 'signup_date' in name.lower()]
print(f"\nFeatures mentioning signup_date: {len(signup_features)}")
for f in signup_features:
    print(f"  {f}")

# Output:
#   Features mentioning signup_date: 0
# ← signup_date contributed ZERO features to the DFS matrix
# The 847 features come entirely from numeric/categorical columns
# and cross-table aggregations on transaction amounts and product categories

Fix: cast signup_date to datetime64 before adding to the EntitySet

The fix is a single line: es['customers']['signup_date'] = pd.to_datetime(es['customers']['signup_date']) applied after the EntitySet is constructed but before ft.dfs() is called. After this cast, pandas dtype changes from object to datetime64[ns]. Woodwork detects the new dtype and re-infers the logical type as Datetime. The primitive dispatcher now matches DatetimeDTFeaturizer against the signup_date column and generates the full suite of temporal transform features.

Alternatively, the cast can be applied to the source DataFrame before it is added to the EntitySet with es.add_dataframe(). Both approaches produce identical results. The important constraint is that the cast must happen before ft.dfs() runs, because Woodwork logical type inference occurs at add_dataframe() time and again when ft.dfs() builds the primitive matching table.

# Fix: cast signup_date to datetime64 before DFS
# featuretools 1.28.x, Python 3.11, pandas 2.1.x

import pandas as pd
import featuretools as ft
from sklearn.model_selection import train_test_split
from sklearn.metrics import roc_auc_score
import lightgbm as lgb

# --- Option A: cast on the source DataFrame before add_dataframe() ---
customers_df['signup_date'] = pd.to_datetime(customers_df['signup_date'])
# dtype is now datetime64[ns]; Woodwork will infer Datetime logical type

es_fixed = ft.EntitySet(id='northfield_churn_fixed')

es_fixed = es_fixed.add_dataframe(
    dataframe_name='customers',
    dataframe=customers_df,
    index='customer_id',
    # signup_date enters as datetime64 → Woodwork infers Datetime → DatetimeDTFeaturizer fires
)

es_fixed = es_fixed.add_dataframe(
    dataframe_name='transactions',
    dataframe=transactions_df,
    index='transaction_id',
    time_index='txn_date',
)

es_fixed = es_fixed.add_dataframe(
    dataframe_name='products',
    dataframe=products_df,
    index='product_id',
)

es_fixed = es_fixed.add_relationship('customers', 'customer_id', 'transactions', 'customer_id')
es_fixed = es_fixed.add_relationship('products', 'product_id', 'transactions', 'product_id')

# --- Option B: cast on the existing EntitySet column (if EntitySet already built) ---
# es['customers']['signup_date'] = pd.to_datetime(es['customers']['signup_date'])
# Same effect — Woodwork re-infers logical type to Datetime on assignment

# Verify the logical type is now Datetime before running DFS
print(f"signup_date logical type: "
      f"{type(es_fixed['customers'].ww.columns['signup_date'].logical_type).__name__}")
# signup_date logical type: Datetime  ← DatetimeDTFeaturizer will now fire

# Run DFS with the corrected EntitySet
features_fixed, feature_defs_fixed = ft.dfs(
    entityset=es_fixed,
    target_dataframe_name='customers',
    max_depth=2,
)

print(f"Total DFS features (fixed): {len(feature_defs_fixed)}")
# Total DFS features (fixed): 912  ← was 847; +65 temporal features

# Verify temporal features are now present
for prefix in ['YEAR', 'MONTH', 'DAY', 'WEEKDAY', 'HOUR', 'DIFF']:
    matches = [f.get_name() for f in feature_defs_fixed if prefix in f.get_name()]
    print(f"  {prefix} features: {len(matches)}")
    if matches:
        print(f"    e.g. {matches[0]}")

# Output:
#   YEAR features: 1
#     e.g. YEAR(signup_date)
#   MONTH features: 1
#     e.g. MONTH(signup_date)
#   DAY features: 1
#     e.g. DAY(signup_date)
#   WEEKDAY features: 1
#     e.g. WEEKDAY(signup_date)
#   HOUR features: 1
#     e.g. HOUR(signup_date)
#   DIFF features: 1
#     e.g. DIFF(signup_date)
# Plus 59 additional cross-table temporal features at max_depth=2

# Inspect the top new temporal features
new_feature_names = set(f.get_name() for f in feature_defs_fixed) - \
                    set(f.get_name() for f in feature_defs)
print(f"\nNew temporal features added ({len(new_feature_names)} total):")
temporal_new = sorted([n for n in new_feature_names
                       if any(p in n for p in ['YEAR', 'MONTH', 'DAY', 'WEEKDAY', 'DIFF'])])
for name in temporal_new[:12]:
    print(f"  {name}")
# Output (sample):
#   DAY(signup_date)
#   DIFF(signup_date)
#   HOUR(signup_date)
#   MAX(transactions.DIFF(txn_date))
#   MEAN(transactions.DIFF(txn_date))
#   MIN(transactions.DIFF(txn_date))
#   MONTH(signup_date)
#   STD(transactions.DIFF(txn_date))
#   WEEKDAY(signup_date)
#   YEAR(signup_date)

# Train churn model on the 912-feature DFS matrix
y = customers_df.set_index('customer_id')['churned'].loc[features_fixed.index]
X_train_f, X_test_f, y_train_f, y_test_f = train_test_split(
    features_fixed, y, test_size=0.2, random_state=42, stratify=y
)

model_fixed = lgb.LGBMClassifier(n_estimators=300, random_state=42)
model_fixed.fit(X_train_f, y_train_f)
auc_fixed = roc_auc_score(y_test_f, model_fixed.predict_proba(X_test_f)[:, 1])
print(f"\nDFS model AUC (fixed): {auc_fixed:.4f}")
# DFS model AUC (fixed): 0.8017

# ---- After fix: summary ----
# DFS features (before fix):  847  ← signup_date as object — no temporal primitives
# DFS features (after fix):   912  ← signup_date as datetime64 — +65 temporal features
#
# AUC before fix:  0.68  ← zero temporal DFS features; tenure signal missing
# AUC after fix:   0.80  ← YEAR/MONTH/DAY/WEEKDAY/DIFF features present
# Manual baseline: 0.81  ← hand-crafted tenure features; near parity achieved
#
# ---- Hour breakdown ----
# Triage:     0.5h — AUC 0.68 vs manual LightGBM + tenure features 0.81;
#             customer tenure likely predictive; DFS feature list inspected
# Root cause: 0.5h — no YEAR/MONTH/DIFF features in feature list despite 847 features;
#             signup_date dtype is object not datetime64;
#             Woodwork shows Unknown logical type; DatetimeDTFeaturizer skipped
# Fix:        0.25h — pd.to_datetime(es['customers']['signup_date']) before DFS call
# Validation: 0.25h — 847→912 features; temporal features created;
#             AUC 0.68→0.80; gap vs manual baseline 0.01; 1.5h total

Related retainer posts

Track retainer hours without a client portal

HourTab turns your time-tracker CSV into a public URL your client bookmarks. No login. No portal. CSV in, URL out.

See pricing →