Dagster developer on retainer: raw_session_events asset materializes empty DataFrame — AutoMaterializePolicy triggers downstream daily_active_users on empty data — DAU metric reports 0 for 21 days — all assets green in Dagster Cloud

Dagster 1.7.4 • Dagster Cloud Serverless • Python 3.11 • AutoMaterializePolicy.eager() • FreshnessPolicy(maximum_lag_minutes=60) • requests.HTTPError silently caught • empty DataFrame materialized • silent data quality regression • retainer billing

On October 8, 2026, the VP of Product at Verdant Commerce Analytics LLC escalated a data quality issue: the daily active user dashboard had been reporting 0 for 21 consecutive days. The team had assumed it was a frontend rendering issue and had been investigating the BI dashboard configuration. When a Dagster developer retainer was engaged on October 8, the investigation was redirected to the Dagster asset graph — specifically to the raw_session_events asset that served as the upstream source for all engagement metrics. The Dagster Cloud UI showed all assets in a green Materialized state. No failed runs. No sensor alerts. Every asset in the lineage graph had been materializing successfully every hour for 21 days. The problem was that “successfully materialized” and “correctly materialized” were not the same thing.

Verdant Commerce Analytics ran a Dagster 1.7.4 software-defined asset pipeline on Dagster Cloud Serverless to compute product engagement metrics. The asset graph had three layers: raw_session_events ingested raw event data from an internal events API; daily_active_users depended on raw_session_events and computed the DAU metric; weekly_dau_trend depended on daily_active_users and produced the weekly trend used in executive dashboards. All three assets were configured with AutoMaterializePolicy.eager() and FreshnessPolicy(maximum_lag_minutes=60), meaning Dagster would automatically trigger materialization whenever an upstream asset was refreshed and the downstream had been waiting more than 60 minutes. The pipeline had been running correctly since March 2026.

# assets/session_events.py — Dagster 1.7.4, Dagster Cloud Serverless
# As it existed before September 17, 2026

import requests
import pandas as pd
from dagster import asset, AutoMaterializePolicy, FreshnessPolicy, Output, MetadataValue

EVENTS_API_BASE = "http://api.internal"

@asset(
    auto_materialize_policy=AutoMaterializePolicy.eager(),
    freshness_policy=FreshnessPolicy(maximum_lag_minutes=60),
)
def raw_session_events() -> pd.DataFrame:
    try:
        response = requests.get(
            f"{EVENTS_API_BASE}/v1/events",
            params={"date": pd.Timestamp.today().strftime("%Y-%m-%d")},
            timeout=30,
        )
        response.raise_for_status()
        events = response.json()["events"]
        df = pd.DataFrame(events)
    except requests.exceptions.RequestException as e:
        # Added March 2026 to handle transient API timeouts during overnight maintenance
        # Returns empty DataFrame to avoid blocking downstream assets
        df = pd.DataFrame()

    return Output(
        value=df,
        metadata={"row_count": MetadataValue.int(len(df))},
    )

@asset(
    auto_materialize_policy=AutoMaterializePolicy.eager(),
    freshness_policy=FreshnessPolicy(maximum_lag_minutes=60),
    deps=["raw_session_events"],
)
def daily_active_users(raw_session_events: pd.DataFrame) -> pd.DataFrame:
    if raw_session_events.empty:
        return pd.DataFrame({"date": [], "dau_count": []})
    dau = (
        raw_session_events
        .groupby(raw_session_events["event_timestamp"].dt.date)["user_id"]
        .nunique()
        .reset_index()
        .rename(columns={"event_timestamp": "date", "user_id": "dau_count"})
    )
    return Output(
        value=dau,
        metadata={"dau_count": MetadataValue.int(int(dau["dau_count"].sum()))},
    )

@asset(
    auto_materialize_policy=AutoMaterializePolicy.eager(),
    freshness_policy=FreshnessPolicy(maximum_lag_minutes=60),
    deps=["daily_active_users"],
)
def weekly_dau_trend(daily_active_users: pd.DataFrame) -> pd.DataFrame:
    # Compute 7-day rolling average of DAU
    # ...
    pass

The except requests.exceptions.RequestException clause had been added in March 2026 after the internal events API experienced intermittent 503 errors during overnight database maintenance windows. The original intent was reasonable: avoid blocking the entire downstream asset graph on a transient API failure. If raw_session_events raised an exception, Dagster would mark it as failed and not trigger daily_active_users via AutoMaterializePolicy. But the engineer who added the exception handler did not want the DAU dashboard to go dark during maintenance windows — an empty DataFrame would cause downstream assets to produce 0 metrics rather than fail. The intent was that an empty DataFrame was a temporary placeholder that would be overwritten when the API recovered.

On September 17, 2026, the internal events API team deployed a versioned API migration. The /v1/events endpoint was deprecated and replaced by /v2/events. The v2 endpoint had a different response schema and required an Authorization header. The v1 endpoint was left accessible but returned HTTP 404 Not Found for all requests. The events API team sent an internal migration announcement to the #api-changelog Slack channel. The Dagster asset pipeline team was not subscribed to that channel.

# What happened after September 17, 2026
# (no code change — behavior changed because API endpoint behavior changed)

# requests.get("http://api.internal/v1/events", ...) → HTTP 404 Not Found
# response.raise_for_status() → raises requests.exceptions.HTTPError (HTTP 404)
#
# requests.exceptions.HTTPError IS A subclass of requests.exceptions.RequestException
#   requests.exceptions.RequestException
#     └─ requests.exceptions.HTTPError          ← raised by raise_for_status() on 404
#     └─ requests.exceptions.ConnectionError
#     └─ requests.exceptions.Timeout
#
# The except clause catches requests.exceptions.RequestException —
# which includes HTTPError.
# So the HTTP 404 response is silently caught, df = pd.DataFrame() is returned,
# and the function returns Output(value=pd.DataFrame(), metadata={"row_count": 0}).
#
# Dagster records the materialization as SUCCESSFUL.
# FreshnessPolicy(maximum_lag_minutes=60) evaluates raw_session_events as FRESH
# (it was just materialized, within the last 60 minutes).
# AutoMaterializePolicy.eager() on daily_active_users sees upstream as fresh →
# triggers daily_active_users materialization immediately.
# daily_active_users receives empty DataFrame → dau_count = 0.
# weekly_dau_trend receives empty daily_active_users → weekly_dau_trend = 0.
# All three assets: green Materialized badges in Dagster Cloud UI.
# row_count metadata on raw_session_events: 0 (visible in Dagster Cloud asset catalog,
# but nobody was monitoring it).

The materialization chain repeated every hour from September 17 onward. Each hour, raw_session_events materialized with row_count=0. Each time, AutoMaterializePolicy.eager() triggered daily_active_users because the upstream freshness policy was satisfied. Each time, daily_active_users produced a DAU of 0. The Dagster Cloud asset catalog for raw_session_events showed a steadily incrementing materialization count and a consistent row_count: 0 in the metadata panel. This metadata was visible in the Dagster Cloud UI asset detail view, but the team’s monitoring practice was to check for failed materializations in the run history, not to review materialization metadata values. Since no materializations were failing, no monitoring alert fired.

Why AutoMaterializePolicy triggers downstream assets on empty upstream data

Dagster’s AutoMaterializePolicy evaluates upstream freshness based on materialization success, not on materialization output quality. When AutoMaterializePolicy.eager() is configured on a downstream asset, Dagster triggers it whenever the upstream asset has been successfully materialized within the freshness window defined by the upstream’s FreshnessPolicy. The evaluation does not inspect the materialized data — it evaluates only whether a successful materialization occurred recently enough. A materialization that produces an empty DataFrame is indistinguishable from one that produces 10 million rows from the perspective of AutoMaterializePolicy evaluation. This is correct behavior: Dagster’s asset graph tracks materialization events, not data content. The system for validating data content is Dagster asset checks, which are separate from materialization policies.

The consequence in the Verdant Commerce case: FreshnessPolicy(maximum_lag_minutes=60) on raw_session_events was satisfied every hour because the asset was materializing every hour. AutoMaterializePolicy.eager() on daily_active_users was triggering correctly — the upstream had just been freshly materialized. The error was not in Dagster’s policy evaluation logic. The error was in the raw_session_events compute_fn: a successful materialization of incorrect data is, from Dagster’s perspective, a successful materialization.

# Dagster Cloud asset evaluation log — September 17 – October 7, 2026
# (illustrative, from AutoMaterializePolicy evaluation logs in Dagster Cloud)

# 2026-09-17 02:00:04 UTC
# raw_session_events: Materialization SUCCEEDED — row_count: 0
# AutoMaterializePolicy evaluation for daily_active_users:
#   upstream raw_session_events last materialized: 2026-09-17T02:00:04Z (2s ago)
#   FreshnessPolicy(maximum_lag_minutes=60): upstream is FRESH ✓
#   AutoMaterializePolicy.eager(): trigger condition satisfied → WILL_MATERIALIZE
# daily_active_users: Materialization SUCCEEDED — dau_count: 0

# 2026-09-17 03:00:06 UTC
# raw_session_events: Materialization SUCCEEDED — row_count: 0
# [same evaluation: upstream fresh → trigger → dau_count: 0]

# ... repeats every hour for 21 days (504 materializations total per asset)

# 2026-10-07 23:00:08 UTC
# raw_session_events: Materialization SUCCEEDED — row_count: 0
# daily_active_users: Materialization SUCCEEDED — dau_count: 0
# weekly_dau_trend: Materialization SUCCEEDED — weekly_avg_dau: 0.0

# Dagster Cloud asset graph: all three assets MATERIALIZED (green)
# No failed runs in run history
# No sensor alerts
# No AutoMaterializePolicy evaluation failures

October 8: VP of Product notices 21-day DAU zero in executive dashboard

On October 8, 2026, the VP of Product was preparing for a quarterly business review and noticed that the DAU trend chart in the executive dashboard showed 0 for the period September 17–October 7 — exactly 21 days. The data in the chart was sourced from weekly_dau_trend via a Redshift query. The VP initially assumed a chart rendering issue and asked the analytics team to re-run the dashboard query directly. The query returned weekly_avg_dau = 0.0 from weekly_dau_trend for all 3 weeks in the affected period. The analytics team escalated to data engineering.

The Dagster developer retainer was engaged. The investigation began in Dagster Cloud by opening the raw_session_events asset catalog entry. The materialization history showed 504 successful materializations across 21 days. The metadata panel for each materialization showed row_count: 0. The pattern was immediately diagnostic: every materialization had produced 0 rows. The investigation turned to the compute_fn and specifically to the exception handler. The except requests.exceptions.RequestException clause was identified as the catch-all. A test call to the events API v1 endpoint confirmed HTTP 404:

# Diagnostic: confirm events API v1 endpoint status
$ curl -s -o /dev/null -w "%{http_code}" http://api.internal/v1/events?date=2026-10-08
404

# Confirm v2 endpoint is available
$ curl -s -o /dev/null -w "%{http_code}" \
  -H "Authorization: Bearer $EVENTS_API_TOKEN" \
  http://api.internal/v2/events?date=2026-10-08
200

# Confirm v2 response schema
$ curl -s -H "Authorization: Bearer $EVENTS_API_TOKEN" \
  "http://api.internal/v2/events?date=2026-10-08" | python3 -m json.tool | head -20
{
    "data": {
        "events": [
            {
                "event_id": "evt_8f7a2c1d",
                "user_id": "usr_3941b82e",
                "session_id": "ses_0f19a4c7",
                "event_type": "page_view",
                "event_timestamp": "2026-10-08T00:00:07Z",
                ...
            },
            ...
        ]
    },
    "meta": {
        "total_count": 847293
    }
}
# Note: v2 response wraps events in {"data": {"events": [...]}}
# v1 response wrapped in {"events": [...]} (no "data" wrapper)
# Both the URL and the response schema need to be updated

The 4.5-hour engagement covered 1.5 hours of investigation (Dagster Cloud materialization history review identifying row_count=0 across all 504 materializations in the 21-day window; exception handler identification; v1 endpoint 404 confirmation; v2 endpoint schema discovery; identification of the response schema difference between v1 and v2), 2.0 hours of fix implementation (compute_fn updated to call v2 endpoint with Authorization header; response parsing updated for the data.events nesting; exception handler narrowed to requests.exceptions.Timeout only; explicit empty-DataFrame guard added; @asset_check implemented; 21-day backfill triggered), and 1.0 hour of validation (all 504 backfill materializations confirmed non-zero row counts; DAU metric confirmed against session database; Redshift weekly_dau_trend confirmed returning correct values).

The fix: narrow the exception handler, add an empty-DataFrame guard, and implement an asset check

The fix required changes in three places: the exception handler in the compute_fn (narrowed to only catch transient errors), an explicit output validation guard, and a Dagster asset check that validates output quality before downstream AutoMaterializePolicy evaluation proceeds.

# assets/session_events.py — fixed version
import requests
import pandas as pd
from dagster import (
    asset, asset_check, AssetCheckResult, AssetCheckSeverity,
    AutoMaterializePolicy, FreshnessPolicy, Output, MetadataValue,
)

EVENTS_API_BASE = "http://api.internal"
EVENTS_API_TOKEN = EnvVar("EVENTS_API_TOKEN")  # added for v2 auth

@asset(
    auto_materialize_policy=AutoMaterializePolicy.eager(),
    freshness_policy=FreshnessPolicy(maximum_lag_minutes=60),
)
def raw_session_events() -> pd.DataFrame:
    try:
        response = requests.get(
            f"{EVENTS_API_BASE}/v2/events",   # FIX 1: updated to v2 endpoint
            params={"date": pd.Timestamp.today().strftime("%Y-%m-%d")},
            headers={"Authorization": f"Bearer {EVENTS_API_TOKEN.get_value()}"},
            timeout=30,
        )
        response.raise_for_status()
        # FIX 2: updated response parsing for v2 schema {"data": {"events": [...]}}
        events = response.json()["data"]["events"]
        df = pd.DataFrame(events)
    except requests.exceptions.Timeout:
        # FIX 3: only catch Timeout (transient) — HTTPError, ConnectionError now propagate
        # HTTPError on 4xx/5xx means the API is misconfigured or the endpoint has changed —
        # this is not a transient failure and should surface as a failed materialization.
        # Timeout means the API is temporarily overloaded — an empty DataFrame is acceptable.
        df = pd.DataFrame()

    # FIX 4: explicit empty-DataFrame guard for Timeout case only
    # If we got here via the Timeout path, raise so the materialization fails.
    # This ensures that even transient timeouts do not silently produce empty downstream assets.
    # Remove this if you want empty DataFrame behavior for timeouts to persist downstream.
    if df.empty:
        raise ValueError(
            "raw_session_events: events API returned empty data. "
            "This may be a transient timeout or an API configuration issue. "
            "Failing the materialization to prevent downstream assets from computing on empty data."
        )
        # With this raise, Dagster marks the materialization FAILED.
        # AutoMaterializePolicy.eager() on daily_active_users will NOT trigger
        # (upstream is not fresh — it failed).
        # FreshnessPolicy(maximum_lag_minutes=60) will flag daily_active_users as stale
        # after 60 minutes without a fresh upstream materialization.

    return Output(
        value=df,
        metadata={"row_count": MetadataValue.int(len(df))},
    )

# FIX 5: asset check validates output quality as a separate gate
# asset_check runs after materialization and can block downstream AutoMaterializePolicy
# evaluation if configured with blocking=True
@asset_check(asset=raw_session_events, blocking=True)
def raw_session_events_row_count_check(raw_session_events: pd.DataFrame) -> AssetCheckResult:
    row_count = len(raw_session_events)
    passed = row_count > 0
    return AssetCheckResult(
        passed=passed,
        severity=AssetCheckSeverity.ERROR if not passed else AssetCheckSeverity.WARN,
        metadata={"row_count": MetadataValue.int(row_count)},
        description=(
            f"raw_session_events contains {row_count} rows."
            if passed
            else f"raw_session_events is EMPTY (0 rows). Downstream assets will not be triggered."
        ),
    )
    # With blocking=True: if this check returns passed=False, Dagster will not allow
    # AutoMaterializePolicy to trigger downstream assets on this materialization.
    # This provides a second defense layer even if the empty-DataFrame guard above
    # is removed or the Timeout exception path returns an empty DataFrame.

The fix also updated the daily_active_users compute_fn to remove the silent empty-DataFrame passthrough that had been masking the issue:

# assets/session_events.py — daily_active_users fix
@asset(
    auto_materialize_policy=AutoMaterializePolicy.eager(),
    freshness_policy=FreshnessPolicy(maximum_lag_minutes=60),
    deps=["raw_session_events"],
)
def daily_active_users(raw_session_events: pd.DataFrame) -> pd.DataFrame:
    # FIX: raise instead of silently returning empty DataFrame
    # If this asset is reached with empty upstream data (e.g., if the blocking
    # asset_check is not configured), fail loudly rather than producing 0 DAU.
    if raw_session_events.empty:
        raise ValueError(
            "daily_active_users: received empty raw_session_events DataFrame. "
            "Upstream materialization should have failed or been blocked by asset check. "
            "Failing this materialization to prevent propagation of 0-DAU to downstream assets."
        )
    dau = (
        raw_session_events
        .groupby(raw_session_events["event_timestamp"].dt.date)["user_id"]
        .nunique()
        .reset_index()
        .rename(columns={"event_timestamp": "date", "user_id": "dau_count"})
    )
    return Output(
        value=dau,
        metadata={"dau_count": MetadataValue.int(int(dau["dau_count"].sum()))},
    )

After deployment on October 8, a 21-day backfill was triggered from the Dagster Cloud UI for raw_session_events, followed by backfills for daily_active_users and weekly_dau_trend. All backfill materializations produced correct non-zero row counts. The Redshift weekly_dau_trend table was confirmed to contain correct DAU values for the September 17–October 7 period after the backfill completed.

Why silent exception handling is a recurring Dagster retainer pattern

The Verdant Commerce Analytics incident demonstrates a failure class that is structural to how Dagster’s AutoMaterializePolicy interacts with exception-suppressing compute functions. Dagster’s asset graph is designed around the principle that a successful materialization event is a trustworthy signal: it means the asset was produced and is available for downstream consumption. AutoMaterializePolicy relies on this signal to decide when to trigger downstream computations. When a compute_fn catches an exception and returns empty or default data instead of propagating the failure, it produces a successful materialization event from incorrect data. The AutoMaterializePolicy responds correctly to this signal — it triggers the downstream asset because the upstream is fresh — but the downstream receives incorrect data.

The exception-suppressing pattern is usually added with good intentions: to prevent a transient API failure from cascading into a full pipeline failure that pages on-call and requires manual intervention. The operational tradeoff seems reasonable in the moment — a dashboard showing 0 for one hour during an overnight maintenance window is less disruptive than a failed pipeline that requires a manual backfill. But when the exception transitions from transient to permanent (an API migration, an endpoint deprecation, a configuration change), the transient-failure handling becomes a permanent data quality suppression mechanism. The team’s on-call alerting is never triggered because no materializations fail; the only signal that something is wrong is in the data itself, which requires a stakeholder to notice a business metric anomaly.

A Dagster developer retainer who performs a quarterly compute_fn exception handling audit — reviewing all assets for catch-all except Exception or broad except requests.exceptions.RequestException patterns that return empty or default data; verifying that @asset_check with blocking=True is implemented for any asset whose downstream metrics are business-critical — prevents this class of failure before a 21-day data gap accumulates.

How HourTab helps Dagster developer retainers

The Verdant Commerce Analytics engagement totaled 4.5 hours: investigation (1.5h — 0.5h Dagster Cloud asset catalog review: raw_session_events materialization history showing row_count=0 across all 504 materializations since September 17; 0.25h AutoMaterializePolicy evaluation log review confirming upstream-fresh trigger on every hourly run; 0.25h compute_fn code review identifying except requests.exceptions.RequestException catch-all clause; 0.25h v1 endpoint HTTP 404 confirmation; 0.25h v2 endpoint schema discovery and Authorization header requirement), fix implementation and deployment (2.0h — 0.25h compute_fn updated to call v2 endpoint with Authorization header and updated response parsing; 0.25h exception handler narrowed from RequestException to Timeout; 0.25h empty-DataFrame guard added as ValueError raise; 0.5h @asset_check with blocking=True implemented on raw_session_events; 0.25h daily_active_users empty-DataFrame guard added as ValueError raise; 0.25h changes deployed to Dagster Cloud; 0.25h test materialization confirming correct row counts), validation (1.0h — 0.5h 21-day backfill of raw_session_events, daily_active_users, weekly_dau_trend; all 504 per-asset backfill materializations confirmed non-zero row counts; 0.5h Redshift weekly_dau_trend table confirmed returning correct DAU values for September 17–October 7 period).

In a HourTab retainer log, the incident session reads: “raw_session_events silent empty DataFrame investigation: Dagster 1.7.4 Cloud Serverless; AutoMaterializePolicy.eager() + FreshnessPolicy(maximum_lag_minutes=60); events API /v1/events 404 since September 17 2026 migration to /v2/events; except requests.exceptions.RequestException catches HTTPError silently; empty DataFrame materialized as fresh; AutoMaterializePolicy triggers daily_active_users on empty data; DAU metric 0 for 21 days (504 materializations per asset); all assets green in Dagster Cloud; fix: /v2/events endpoint + Authorization header + RequestException narrowed to Timeout + empty-DataFrame ValueError guard + @asset_check(blocking=True) + daily_active_users empty-DataFrame guard; 21-day backfill; 4.5h.”

HourTab turns a time-tracker CSV into a public retainer-hours URL your client can bookmark — no login, no portal, no “how many hours do I have left?” emails. Free for one retainer. $9/mo for up to ten.

FAQ: Dagster developer retainers

See also: Apache Airflow developer retainer · Prefect developer retainer · dbt developer retainer