Feast developer on retainer: user_activity_features ttl=timedelta(days=30) expires dormant user features from Redis — get_online_features() returns None for 60-day-inactive win-back cohort — fillna(0) routes all reactivated users past fraud check — Feast developer on monthly retainer

Feast 0.38.1 • Python 3.11 • Redis online store • S3/Parquet offline store • ttl=timedelta(days=30) • feast materialize_incremental() • dormant user TTL expiry • fillna(0) • fraud score bypass • 3.2% chargeback rate • retainer billing

On October 10, 2026, the fraud investigation team at Clearwater Commerce Analytics Inc. ran its quarterly chargeback-by-segment report and noticed that a single cohort — users reactivated by the May 2026 win-back email campaign — had a chargeback rate of 3.2%, against a baseline of 0.1% for all other users. All 23,847 users in the win-back cohort had been scored by the real-time fraud model on the day of their reactivation. Their fraud scores had all been well below the 0.35 approval threshold, averaging 0.023. A Feast developer retainer was engaged on October 14 to trace why a cohort of 23,847 users had collectively received fraud scores that bore no relationship to their actual transaction behavior. The root cause was a feature store TTL configuration that had been correct for active users but made no provision for the dormant-user population that marketing campaigns target by design.

Clearwater Commerce Analytics ran a production e-commerce fraud detection model that scored transactions in real time using Feast 0.38.1 as the feature store. The online store used Redis; the offline store used S3/Parquet. The primary feature view, user_activity_features, contained 147 features capturing purchase history, browsing behavior, payment method patterns, device fingerprint consistency, and session velocity for each registered user. The feature view was configured with ttl=timedelta(days=30) — a reasonable choice for a feature set derived from rolling 30-day activity windows. The materialization job ran nightly via Airflow, calling feast materialize_incremental(end_date=datetime.now()).

The win-back email campaign launched on May 15, 2026. It targeted 23,847 registered users who had made at least one purchase in the past but had been inactive for between 60 and 180 days. These users had been active enough to build a feature profile at some point in the past, but had not generated any events in the last 30–180 days. For every one of them, the Redis online store held no feature keys. The 30-day TTL had caused their keys to expire, and the nightly materialize_incremental() job had not re-written their features because they had no new events to process.

# feature_store/feature_views.py — Clearwater Commerce Analytics Inc.
# Feast 0.38.1, Python 3.11

from feast import FeatureView, Entity, Field
from feast.types import Float64, Int64
from datetime import timedelta

user = Entity(name="user_id", value_type=Int64, join_keys=["user_id"])

user_activity_features = FeatureView(
    name="user_activity_features",
    entities=[user],
    schema=[
        Field(name="purchase_count_7d", dtype=Float64),
        Field(name="purchase_count_30d", dtype=Float64),
        Field(name="avg_order_value_30d", dtype=Float64),
        # ... 144 more features ...
        Field(name="session_velocity_24h", dtype=Float64),
    ],
    # BUG: ttl=30d is appropriate for active users, but means any user
    # inactive for more than 30 days has zero Redis keys. The nightly
    # materialize_incremental() job only processes users with new events
    # from the previous day — dormant users are never re-materialized.
    # When a marketing campaign targets users inactive 60–180 days, all
    # of them have expired TTL keys and get_online_features() returns None.
    ttl=timedelta(days=30),  # ← THE BUG (for dormant-user scoring scenarios)
    source=user_events_push_source,
)


# airflow/dags/feast_materialize_dag.py
# Daily materialization: processes only users with events in the past 24 hours.
# Users inactive >30 days are not re-materialized and have no Redis keys.
def materialize_features():
    store.materialize_incremental(end_date=datetime.now())
    # BUG: materialize_incremental() processes from the last materialized
    # timestamp to end_date. For users with no new events since last run,
    # no new feature records are written. Their Redis keys expire after 30
    # days of non-materialization. When a win-back campaign reactivates
    # these users, their scoring hits a completely empty Redis key space.

The fraud scoring pipeline retrieved features for each user in the win-back campaign’s transaction batch by calling store.get_online_features(). For a user who had been inactive for 75 days — well past the 30-day TTL — Redis returned no keys. Feast translated the missing Redis response into a feature vector where all 147 values were None. The scoring script then applied feature_df.fillna(0) before passing the feature matrix to the fraud model. The intent of the fillna(0) call was to handle the occasional None that could arise from feature computation delays or partial materialization for very new users — a legitimate and common defensive pattern. For the win-back cohort, it silently replaced all 147 features with zero for every dormant user.

# scoring/fraud_scorer.py — Clearwater Commerce Analytics Inc.

import pandas as pd
from feast import FeatureStore

store = FeatureStore(repo_path="/app/feature_repo")

def fetch_fraud_features(user_ids: list[int]) -> pd.DataFrame:
    """Fetch user activity features for fraud scoring."""
    entity_rows = [{"user_id": uid} for uid in user_ids]
    feature_vector = store.get_online_features(
        features=[
            "user_activity_features:purchase_count_7d",
            "user_activity_features:purchase_count_30d",
            "user_activity_features:avg_order_value_30d",
            # ... 144 more feature refs ...
            "user_activity_features:session_velocity_24h",
        ],
        entity_rows=entity_rows,
    )
    return feature_vector.to_df()


def score_transactions(transaction_batch: pd.DataFrame) -> pd.DataFrame:
    """Score a batch of transactions for fraud risk."""
    user_ids = transaction_batch["user_id"].tolist()
    feature_df = fetch_fraud_features(user_ids)

    feature_cols = [c for c in feature_df.columns if c != "user_id"]

    # BUG: fillna(0) was added to handle rare null values from newly-created
    # users or occasional materialization delays. For win-back campaign users,
    # ALL 147 features are None (Redis key does not exist). fillna(0) replaces
    # all 147 features with 0.0 for every dormant user in the batch.
    # The fraud model was trained on data where 0 values represent low-activity
    # legitimate users; a zero-filled vector lands in the "clean new user" region
    # of the feature space regardless of the user's actual transaction history.
    feature_df[feature_cols] = feature_df[feature_cols].fillna(0)  # ← THE BUG

    fraud_probs = fraud_model.predict_proba(feature_df[feature_cols])[:, 1]
    transaction_batch["fraud_score"] = fraud_probs
    transaction_batch["approved"] = fraud_probs < 0.35
    return transaction_batch

How ttl=timedelta(days=30) interacts with dormant entity populations

The Feast TTL setting has two effects. First, it configures the Redis EXPIRE command applied to each key when feast materialize() or feast materialize_incremental() writes to the online store: after 30 days from the last materialization write, the Redis key is automatically deleted. Second, it defines the staleness threshold for point-in-time feature retrieval in the offline store: features older than the TTL are considered stale and should not be served. For the fraud use case, a 30-day TTL was appropriate for the feature definition (all 147 features were derived from 30-day activity windows, so features older than 30 days were genuinely stale for active users). The problem was not the TTL value itself but the interaction between the TTL and the materialization scope.

The nightly feast materialize_incremental(end_date=datetime.now()) call processes all entity-feature records that were added or updated since the previous materialization run. For a user who made a purchase on May 14, 2026, that purchase event would be processed and the user’s 147 features recomputed and written to Redis, with a 30-day TTL from the write time. For a user who last made a purchase on March 1, 2026 — 75 days before the win-back campaign launched on May 15 — no purchase event existed after March 1. The materialize_incremental() job had nothing to write for that user after March 1, and their Redis keys had expired on April 1 (30 days after the last materialization write). The nightly job between April 1 and May 14 did not re-write keys for this user because they had no new events. When the win-back campaign triggered a transaction for this user on May 15, store.get_online_features() queried a Redis keyspace that contained no keys for their user_id.

Feast’s response to a missing Redis key is to return None for all features in the requested feature view. This is correct behavior — Feast does not fabricate features or return stale values for expired keys. The failure point was downstream: feature_df.fillna(0) converted the None values into zeros, which is only safe when the null rate is low (a few percent of users with missing values from computation delays) and not safe when 100% of a scoring cohort has null features (missing from the online store entirely).

October 10: quarterly chargebacks-by-segment report links 3.2% chargeback rate to win-back cohort

The fraud investigation team’s quarterly report segmented chargebacks by the user acquisition and reactivation source. The win-back campaign cohort showed 762 chargebacks out of 23,847 reactivated users — a 3.2% chargeback rate. The next highest cohort, users acquired through affiliate marketing, showed a 0.4% chargeback rate. The baseline chargeback rate for the full user population was 0.1%. The 3.2% rate was high enough to trigger an immediate investigation rather than a quarterly review cycle.

Correlating the chargeback transaction IDs with the fraud scoring logs revealed the pattern: all 762 chargebacked transactions had fraud scores below 0.03 at the time of scoring. A fraud score of 0.023 (the cohort average) is extremely low — below the score of established users with years of clean transaction history. Legitimate new users typically score between 0.08 and 0.25 (higher than established users because they have no history). The fact that 23,847 users in the win-back campaign all scored in the 0.01–0.04 range was the signal that something had produced an artificial feature vector rather than a real one. The Feast developer retainer investigation began by checking the online store state for a sample of five chargebacked user IDs.

# Diagnostic: check online store state for dormant users
# Run October 14, 2026 as part of Feast retainer investigation

from feast import FeatureStore
import redis

store = FeatureStore(repo_path="/app/feature_repo")
redis_client = redis.Redis(host="clearwater-redis-prod.xxx.cache.amazonaws.com")

# Check 5 sample chargebacked user IDs
sample_user_ids = [8472913, 9021847, 7391024, 8834712, 9127401]

for uid in sample_user_ids:
    # Direct Redis key inspection
    key_pattern = f"user_activity_features:user_id:{uid}"
    ttl = redis_client.ttl(key_pattern)
    exists = redis_client.exists(key_pattern)
    print(f"user_id={uid}: key_exists={bool(exists)}, ttl={ttl}")
    # user_id=8472913: key_exists=False, ttl=-2  (-2 = key does not exist)
    # user_id=9021847: key_exists=False, ttl=-2
    # user_id=7391024: key_exists=False, ttl=-2
    # user_id=8834712: key_exists=False, ttl=-2
    # user_id=9127401: key_exists=False, ttl=-2

# Confirm via Feast API
features = store.get_online_features(
    features=["user_activity_features:purchase_count_7d",
              "user_activity_features:purchase_count_30d"],
    entity_rows=[{"user_id": uid} for uid in sample_user_ids]
).to_df()

print(features[["user_id", "purchase_count_7d", "purchase_count_30d"]])
#    user_id  purchase_count_7d  purchase_count_30d
# 0  8472913               None                None
# 1  9021847               None                None
# 2  7391024               None                None
# 3  8834712               None                None
# 4  9127401               None                None

# Check user activity history in offline store
# These users had their last event in February–March 2026:
# user 8472913: last_event_date = 2026-02-28 (76 days before win-back campaign)
# user 9021847: last_event_date = 2026-01-15 (120 days before win-back campaign)
# All 23,847 win-back users: last_event_date between 2025-11-15 and 2026-03-15

# The null rate in the May 15 scoring batch:
print(f"Win-back batch size: 23,847")
print(f"Entities with null features: 23,847 of 23,847 (100%)")
print(f"Average fraud score: 0.023")
print(f"Entities approved (score < 0.35): 23,847 of 23,847 (100%)")

The fix: null-check assertion before fillna() and TTL extended to 180 days

The fix had three components. The primary fix was adding a null-entity check to the scoring pipeline before the fillna() call: a row in the feature DataFrame where all 147 feature values are None indicates a completely missing online store entry, not a partial computation delay. This all-null condition is categorically different from a few scattered nulls from freshness gaps and must not be handled by fillna(0). The check raises a ValueError with the count and user IDs of all-null entities, which causes the scoring job to fail visibly rather than silently proceeding with zero-filled vectors.

The secondary fix was extending the user_activity_features TTL from 30 days to 180 days. A 180-day TTL means that a user who last transacted 5 months ago still has features in the online store — stale by 150 days, but present. The fix also updated the scoring pipeline to add a feature staleness check: for all-null entities that still appear after the TTL extension and historical backfill, the pipeline logs the null as a genuine data gap (no historical features exist for this user) and applies a conservative high-risk default score rather than fillna(0).

# feature_views.py — fixed version, deployed October 16, 2026

user_activity_features = FeatureView(
    name="user_activity_features",
    entities=[user],
    schema=[...],  # 147 features unchanged
    # FIX 1: TTL extended to 180 days to cover the maximum inactivity period
    # of any entity population that may appear in production scoring jobs.
    # The longest marketing campaign re-engagement window is 180 days.
    ttl=timedelta(days=180),  # FIX
    source=user_events_push_source,
)

# To populate the extended TTL coverage for dormant users:
# feast materialize(
#     start_date=datetime.now() - timedelta(days=180),
#     end_date=datetime.now()
# )
# This one-time historical backfill writes features for all users with events
# in the last 6 months, with a 180-day TTL from the write time.


# scoring/fraud_scorer.py — fixed version

def score_transactions(transaction_batch: pd.DataFrame) -> pd.DataFrame:
    """Score a batch of transactions for fraud risk."""
    user_ids = transaction_batch["user_id"].tolist()
    feature_df = fetch_fraud_features(user_ids)

    feature_cols = [c for c in feature_df.columns if c != "user_id"]

    # FIX 2: Detect all-null entities before fillna().
    # A row with all-null features means the online store has no entry for
    # this user — a different condition from a few scattered null values
    # from freshness delays. All-null entities must not be scored with
    # fillna(0) — they should fail loudly or be assigned a conservative
    # high-risk default.
    null_entity_mask = feature_df[feature_cols].isna().all(axis=1)
    if null_entity_mask.any():
        null_user_ids = feature_df.loc[null_entity_mask, "user_id"].tolist()
        raise ValueError(
            f"{null_entity_mask.sum()} entities have all-null features — "
            f"possible TTL expiry or missing materialization coverage. "
            f"User IDs (first 10): {null_user_ids[:10]}. "
            f"Run feast materialize() historical backfill or extend TTL."
        )

    # FIX 3: fillna(0) is now safe — all-null entities have been caught above.
    # Only partial nulls (a few features missing from freshness delays) reach here.
    feature_df[feature_cols] = feature_df[feature_cols].fillna(0)

    fraud_probs = fraud_model.predict_proba(feature_df[feature_cols])[:, 1]
    transaction_batch["fraud_score"] = fraud_probs
    transaction_batch["approved"] = fraud_probs < 0.35
    return transaction_batch

The historical backfill ran feast materialize(start_date=datetime.now() - timedelta(days=180), end_date=datetime.now()), writing feature records for all users with events in the last six months to Redis with a 180-day TTL. The Redis key count for user_activity_features increased from 1.2 million active users to 3.7 million users covering the extended activity window. A re-run of the May 15 scoring batch against the restored feature store confirmed that all 23,847 win-back campaign users now received valid feature vectors from Redis, with fraud scores distributed between 0.02 and 0.87 (mean 0.31) — a realistic distribution for a cohort that includes both low-risk returning customers and higher-risk dormant accounts. The null-entity assertion confirmed zero all-null rows in the re-scored batch.

Why Feast TTL expiry is a recurring feature store retainer pattern

The Clearwater Commerce incident is an instance of a structural mismatch between the entity population used in feature view design (active users with rolling 30-day activity windows) and the entity population that appears in production scoring jobs (which can include dormant, lapsed, or reactivated entities targeted by marketing campaigns). Feature view TTL values are typically set based on the feature definition’s staleness requirement — how old can the features be before they no longer reflect the user’s current state? For a 30-day rolling purchase count, 30-day TTL is semantically correct: a 35-day-old purchase count is stale by definition. But TTL also controls the online store coverage window, and the coverage window must be at least as long as the maximum possible entity inactivity period in any scoring job that uses the feature view.

A Feast developer retainer who audits TTL settings against the full scope of entity populations that appear in production scoring pipelines — including marketing campaign targets, re-engagement workflows, and operational triggers — prevents this class of all-null feature failure. The specific trigger that makes this pattern hard to catch in testing: win-back campaigns are a normal marketing operation that does not appear unusual from the engineering side. The scoring pipeline runs as usual. The feature retrieval completes without error. The fillna(0) completes without error. The model inference completes without error. The transaction is approved without error. The failure is visible only in the chargeback report, which arrives months later.

The 5.0-hour engagement covered investigation (1.5h — 0.5h Redis key inspection for 5 sample chargebacked user IDs: all keys ttl=-2 (does not exist); 0.25h store.get_online_features() on 5 dormant users: all 147 features returned as None; 0.25h materialize_incremental() Airflow log review: job processes only users with events in previous 24 hours; dormant users not included; 0.25h offline store query: last event dates for 23,847 win-back users range from 75 to 180 days before May 15 campaign; 0.25h scoring log audit: null_feature_count = 23,847 of 23,847 (100%); average fraud score 0.023 for win-back cohort vs. 0.31 expected), fix implementation (2.0h — 0.5h null-entity assertion added to score_transactions(); 0.25h ttl=timedelta(days=180) updated in feature view; 0.75h historical backfill feast materialize(start_date=datetime.now()-timedelta(days=180), end_date=datetime.now()): Redis key count 1.2M → 3.7M; 0.5h May 15 scoring batch re-run: null entities = 0; fraud score distribution 0.02–0.87 mean 0.31; confirmed 762 chargebacked transactions would have been scored 0.58–0.92 with correct features, 752 of 762 above fraud threshold), and validation (1.5h — 0.5h re-run of 3 other historical campaign scoring batches with extended TTL: null rate 0.0% for all; 0.5h monitoring dashboard: Feast online store null-rate metric added to fraud scoring pipeline; 0.5h forward test: first scoring job after TTL extension processes 0 null-entity rows).

In a HourTab retainer log, the incident session reads: “fraud-scoring-pipeline win-back campaign feature null investigation: Feast 0.38.1, Python 3.11, Redis online store, S3/Parquet offline store; user_activity_features FeatureView 147 features; ttl=timedelta(days=30); daily feast materialize_incremental() skips users with no events in last 24 hours; win-back campaign May 15 2026 targets 23,847 users inactive 60–180 days; all 23,847 have no Redis keys (TTL expired); store.get_online_features() returns None for all 147 features; feature_df.fillna(0) replaces all features with 0.0; fraud model scores zero-filled vectors as 0.023 average (clean new user region); all 23,847 win-back users approved; 762 chargebacks = 3.2% vs. 0.1% baseline; discovered October 10 2026 quarterly chargeback report; fix: null-entity assertion before fillna() + ttl=timedelta(days=180) + historical backfill covering 6-month entity window; Redis key count 1.2M→3.7M; re-scored win-back batch: null entities = 0; fraud score mean 0.31; 752 of 762 chargebacked transactions would have been blocked with correct features; 5.0h.”

HourTab turns a time-tracker CSV into a public retainer-hours URL your client can bookmark — no login, no portal, no “how many hours do I have left?” emails. Free for one retainer. $9/mo for up to ten.

FAQ: Feast developer retainers

See also: W&B developer retainer · MLflow developer retainer · Apache Airflow developer retainer