Great Expectations developer on retainer: product_catalog_checkpoint FAILED for 6 weeks — downstream ETL loads data regardless of result[‘success’] — 1,243 Bulk Supplies products excluded from dim_products — $502K revenue absent from Q3 attribution

Great Expectations (GX) 0.18.15 • Python 3.11 • Amazon S3 datasource • Redshift • dbt • expect_column_values_to_be_between • SlackNotificationAction • checkpoint result not checked • expectation suite staleness • retainer billing

On October 10, 2026, a business analyst at Pinehurst Data Commerce Corp was preparing Q3 revenue attribution numbers and noticed that the Bulk Commercial Supplies product category — which had launched on August 28, 2026 — did not appear in any revenue attribution reports for the entire 6-week period since launch. The products were in the product catalog system. Customers had purchased them. Invoices had been generated. But dim_products, the dimension table used as the source of truth for all revenue attribution in the Redshift data warehouse, contained zero Bulk Commercial Supplies products. When a Great Expectations developer retainer was engaged, the investigation traced the missing $502,481 in Bulk Supplies revenue to three compounding failures: a GX checkpoint that had been FAILED every day for 42 days without anyone noticing, a downstream ETL pipeline that loaded data regardless of whether the checkpoint passed, and a dbt model filter that excluded all products outside the original price range.

Pinehurst Data Commerce Corp ran a daily product catalog ingestion pipeline that read product data from S3 Parquet files, ran a Great Expectations 0.18.15 checkpoint to validate data quality, and loaded the validated data to Redshift. The GX checkpoint had been in place since March 2026, when the data engineering team adopted GX as part of a data quality initiative. The expectation suite product_catalog_expectations included 11 expectations covering column existence, null rates, string length bounds, and value ranges. One of those expectations was expect_column_values_to_be_between("unit_price", min_value=0.01, max_value=999.99), set when the only product categories were consumer electronics and software (all with unit prices between $9.99 and $899.99). The checkpoint’s action_list included a SlackNotificationAction configured to send alerts to #data-quality-logs on checkpoint failure.

# great_expectations/checkpoints/product_catalog_checkpoint.yml
# GX 0.18.15 (YAML-based checkpoint configuration)

name: product_catalog_checkpoint
config_version: 1.0
class_name: Checkpoint
run_name_template: "%Y%m%d-%H%M%S-product-catalog"
validations:
  - batch_request:
      datasource_name: s3_product_catalog
      data_connector_name: configured_data_connector
      data_asset_name: product_catalog_parquet
      data_connector_query:
        index: -1  # latest partition
    expectation_suite_name: product_catalog_expectations

action_list:
  - name: store_validation_result
    action:
      class_name: StoreValidationResultAction
  - name: update_data_docs
    action:
      class_name: UpdateDataDocsAction
  - name: slack_alert
    action:
      class_name: SlackNotificationAction
      slack_webhook: "https://hooks.slack.com/services/T0123/B0456/abcdef"
      notify_on: failure        # sends Slack alert when checkpoint fails
      notify_with:
        - "store_validation_result"

# Channel #data-quality-logs was created March 2026 for GX checkpoint alerts.
# As of August 28, 2026: 0 active subscribers in #data-quality-logs.
# (Engineers who originally subscribed had moved to other teams or left the company.
#  Channel had been silent for 3 months before the Bulk Supplies failure began.)
# Result: Slack alert fires correctly to the channel; nobody sees it.

# great_expectations/expectations/product_catalog_expectations.json (excerpt)
# Created March 2026; never updated after initial setup
{
  "expectation_type": "expect_column_values_to_be_between",
  "kwargs": {
    "column": "unit_price",
    "min_value": 0.01,
    "max_value": 999.99
    # Set March 2026 based on all existing product prices: $9.99 – $899.99
    # Bulk Commercial Supplies (added August 28, 2026): $1,200 – $4,500
    # All 1,243 Bulk Supplies products exceed max_value=999.99
    # This expectation will FAIL on every run from August 28 onward
    # until the expectation suite is updated
  },
  "meta": {}
}

On August 28, 2026, the product catalog team added the Bulk Commercial Supplies category to Pinehurst’s catalog system. The first daily Parquet export to S3 on August 28 included 1,243 new products with unit_price values ranging from $1,200.00 to $4,500.00. When the daily product_catalog_checkpoint ran at 3:00 AM UTC on August 28, expect_column_values_to_be_between("unit_price", min_value=0.01, max_value=999.99) failed: 1,243 of the 89,417 rows in the Parquet file had unit_price values above 999.99. The checkpoint returned a FAILED result. The SlackNotificationAction fired and sent a correctly formatted alert to #data-quality-logs. The Data Docs site was updated to show the FAILED checkpoint run. GX had correctly detected the data quality issue.

# loaders/product_catalog_loader.py — as it existed August 28, 2026
# BUG: checkpoint runs but result is not checked before loading

import great_expectations as gx
import pandas as pd
from sqlalchemy import create_engine
import pyarrow.parquet as pq
import s3fs

def run_daily_load(s3_path: str, redshift_conn_str: str):
    context = gx.get_context()

    # Run the GX checkpoint for data quality validation
    result = context.run_checkpoint(checkpoint_name="product_catalog_checkpoint")

    # BUG: result.success (= False) is never checked.
    # The comment below was added by the original developer to explain the intent.
    # "GE checkpoint runs for observability; load proceeds regardless of result."
    # This comment reflects the actual design decision at the time GX was adopted:
    # the team did not want a data quality failure to block the daily product catalog update,
    # because they expected GX to surface issues that they would then fix retroactively.
    # The consequence: GX is advisory-only. Any data that passes the Redshift COPY
    # will be loaded, regardless of what GX found.

    # Load data to Redshift regardless of checkpoint result
    fs = s3fs.S3FileSystem()
    df = pq.read_table(s3_path, filesystem=fs).to_pandas()
    engine = create_engine(redshift_conn_str)
    df.to_sql(
        "product_catalog",
        engine,
        schema="staging",
        if_exists="replace",
        index=False,
    )
    # Result: 89,417 rows loaded to staging.product_catalog,
    # including all 1,243 Bulk Commercial Supplies products with unit_price > 999.99.
    # Redshift column definition: unit_price DECIMAL(7,2) — max value 99,999.99.
    # All 1,243 products load without error. Data is present in Redshift.
    print(f"Loaded {len(df)} rows to staging.product_catalog")

The downstream issue then compounded the problem. The dbt model that transformed staging.product_catalog into dim_products had a WHERE unit_price <= 999.99 filter that had been added in March 2026 by the analytics engineer who built the model, to mirror the GX expectation range. The intent was to ensure that dim_products only contained products that had passed the GX data quality check. But by implementing this as a static SQL filter rather than as a check against GX validation results, the filter became a permanent exclusion rule that silently dropped any product with a price above $999.99:

-- models/dim_products.sql — dbt model, as of August 28, 2026
-- BUG: WHERE unit_price <= 999.99 silently excludes all Bulk Supplies products

{{ config(materialized='table') }}

SELECT
    product_id,
    product_name,
    product_category,
    sku,
    unit_price,
    cost_price,
    supplier_id,
    is_active,
    created_at,
    updated_at
FROM {{ source('staging', 'product_catalog') }}
WHERE
    is_active = TRUE
    AND unit_price <= 999.99
    -- Filter added March 2026 to mirror GX expect_column_values_to_be_between bounds.
    -- Original intent: exclude any products that would fail the GX price expectation.
    -- Actual consequence: permanently excludes any product priced above $999.99,
    -- including all Bulk Commercial Supplies added August 28, 2026 ($1,200–$4,500).
    AND unit_price > 0

The combined effect: GX detected the data quality issue on day 1 and alerted correctly to an unmonitored channel. The ETL loaded the data into Redshift staging tables correctly. The dbt dim_products model then silently excluded all 1,243 Bulk Supplies products on every daily run from August 28 through October 10 — 42 consecutive dbt runs. Every revenue attribution query that sourced from dim_products therefore excluded Bulk Supplies products from the count, producing revenue attribution reports that were missing an entire product category.

October 10: $502K revenue gap discovered in Q3 close preparation

On October 10, the business analyst noticed that the Bulk Commercial Supplies category — which had been actively marketed since late August — appeared in the invoicing system but not in any revenue attribution report. The analyst checked the Redshift product_catalog staging table directly and confirmed that 1,243 Bulk Supplies products were present with correct unit prices. The analyst then queried dim_products and confirmed that WHERE product_category = 'Bulk Commercial Supplies' returned 0 rows. This discrepancy — products present in staging but absent from dim_products — escalated to the data engineering team.

The Great Expectations developer retainer was engaged on October 10. The investigation began in GX Data Docs. The Data Docs site for product_catalog_checkpoint showed 42 consecutive FAILED checkpoint runs, starting August 28, each with a red badge and an expectation result detail showing 1,243 unexpected values in the unit_price column. The failed expectation was immediately apparent: expect_column_values_to_be_between("unit_price", min_value=0.01, max_value=999.99) with an unexpected_percent of approximately 1.4% (1,243 / 89,417). The Slack alert history in #data-quality-logs confirmed that Slack had sent 42 alert messages to the channel, all of which had been delivered successfully, and none of which had been seen by anyone.

# GX Data Docs — product_catalog_checkpoint run history (excerpt)
# All 42 runs since August 28, 2026: FAILED
# (screenshot confirms 42 red FAILED badges in run list)

# Run detail for 2026-08-28-030012 (first failed run):
# Expectation Suite: product_catalog_expectations
# Status: FAILED
# Successful expectations: 10 / 11
# Failed expectations:     1 / 11
#
# Failed expectation:
#   expect_column_values_to_be_between
#   Column: unit_price
#   min_value: 0.01, max_value: 999.99
#   observed_value: {"min": 0.01, "max": 4500.00}
#   unexpected_count: 1,243
#   unexpected_percent: 1.39%
#   partial_unexpected_list: [1200.00, 1350.00, 2400.00, 4500.00, 1200.00, ...]
#   element_count: 89,417
#   missing_count: 0
#
# Same pattern repeated for all 42 runs: always 1,243 unexpected values,
# always the Bulk Commercial Supplies products, always between $1,200 and $4,500.

# Slack alert (sent to #data-quality-logs, 0 subscribers):
# "🔴 product_catalog_checkpoint FAILED
# Run: 20261010-030008-product-catalog
# Batch: product_catalog_parquet / 2026-10-10
# Expectations: 10 passed, 1 failed
# Failed: expect_column_values_to_be_between (unit_price) — 1,243 unexpected values
# Data Docs: https://pinehurst-gx-docs.s3.amazonaws.com/..."

The investigation confirmed the full causal chain: GX had correctly identified the data quality issue on day 1 and had sent 42 correct Slack alerts, none of which were seen. The ETL pipeline had loaded all 1,243 Bulk Supplies products into Redshift staging on every daily run. The dbt model had excluded them from dim_products on every daily run. The downstream revenue attribution queries had correctly queried dim_products and returned no Bulk Supplies revenue because there were no Bulk Supplies products in dim_products. The missing revenue was $502,481 across 42 days of Bulk Supplies sales that were recorded in the invoicing system but absent from all analytics.

The 5.5-hour engagement covered 1.5 hours of investigation (GX Data Docs review: 42 consecutive FAILED runs confirmed, all with 1,243 unexpected unit_price values; ETL source code review: absence of result[‘success’] check confirmed; dbt dim_products SQL review: WHERE unit_price <= 999.99 confirmed; Slack channel audit: 42 delivered alerts to 0-subscriber channel confirmed; Redshift staging vs. dim_products row count discrepancy confirmed: 1,243 Bulk Supplies in staging, 0 in dim_products), fix implementation and deployment (2.5h: ETL pipeline updated to check result[‘success’] and raise ValueError on failure; GX expectation suite updated from max_value=999.99 to max_value=9999.99 with meta note; dbt dim_products WHERE unit_price <= 999.99 filter removed; Slack channel updated from #data-quality-logs to #data-quality-alerts with 4 active subscribers; changes deployed and a manual checkpoint run confirmed PASSED with 0 unexpected values), and backfill and validation (1.5h: dbt dim_products model re-run to include all 42 days of Bulk Supplies data; 1,243 products confirmed present in dim_products after re-run; revenue attribution re-run confirmed $502,481 Bulk Supplies revenue now present in Q3 attribution totals).

The fix: check result[‘success’], update the expectation, and fix the dbt filter

The fix required changes in three places: the ETL pipeline (check the checkpoint result and halt on failure), the GX expectation suite (update the price range bound to reflect the actual business data), and the dbt model (remove the static price filter that silently excluded valid products):

# loaders/product_catalog_loader.py — fixed version
import great_expectations as gx
import pandas as pd
from sqlalchemy import create_engine
import pyarrow.parquet as pq
import s3fs
import json

def run_daily_load(s3_path: str, redshift_conn_str: str):
    context = gx.get_context()

    # Run the GX checkpoint for data quality validation
    result = context.run_checkpoint(checkpoint_name="product_catalog_checkpoint")

    # FIX: check result.success before proceeding with load
    if not result.success:
        failed_expectations = []
        for validation_result in result.run_results.values():
            for er in validation_result.results:
                if not er.success:
                    failed_expectations.append({
                        "expectation_type": er.expectation_config.expectation_type,
                        "column": er.expectation_config.kwargs.get("column"),
                        "unexpected_count": er.result.get("unexpected_count"),
                        "unexpected_percent": er.result.get("unexpected_percent"),
                    })
        raise ValueError(
            f"product_catalog_checkpoint FAILED. "
            f"Failed expectations: {json.dumps(failed_expectations, indent=2)}. "
            f"Load halted. Review Data Docs for details."
        )
        # With this raise:
        # - The data load step below never runs when the checkpoint fails.
        # - The Airflow DAG running this script marks the task as FAILED.
        # - The on-call Slack alert fires for the DAG failure (not just the GX alert).
        # - The product_catalog staging table is not updated with out-of-range data.
        # - The downstream dbt run is not triggered (DAG stops at this task).

    # Load data only if checkpoint passed
    fs = s3fs.S3FileSystem()
    df = pq.read_table(s3_path, filesystem=fs).to_pandas()
    engine = create_engine(redshift_conn_str)
    df.to_sql(
        "product_catalog",
        engine,
        schema="staging",
        if_exists="replace",
        index=False,
    )
    print(f"Checkpoint PASSED. Loaded {len(df)} rows to staging.product_catalog")

# great_expectations/expectations/product_catalog_expectations.json — updated expectation
{
  "expectation_type": "expect_column_values_to_be_between",
  "kwargs": {
    "column": "unit_price",
    "min_value": 0.01,
    "max_value": 9999.99
    # Updated October 10, 2026: raised max_value from 999.99 to 9999.99
    # to include Bulk Commercial Supplies category ($1,200 – $4,500).
    # Original 999.99 bound was set March 2026 based on consumer electronics
    # and software category prices only.
    # If new product categories with higher prices are added in the future,
    # this expectation must be updated to match the new valid price range.
  },
  "meta": {
    "last_updated": "2026-10-10",
    "reason": "Bulk Commercial Supplies category launch August 28 2026 (unit_price $1,200-$4,500)"
  }
}

# Note: consider replacing static range expectation with a per-category expectation:
# expect_column_pair_values_to_be_in_set or a custom expectation that maps
# product_category to allowed price ranges. This would prevent a future category
# from silently failing if its price range exceeds the global max_value.

-- models/dim_products.sql — fixed version
{{ config(materialized='table') }}

SELECT
    product_id,
    product_name,
    product_category,
    sku,
    unit_price,
    cost_price,
    supplier_id,
    is_active,
    created_at,
    updated_at
FROM {{ source('staging', 'product_catalog') }}
WHERE
    is_active = TRUE
    -- FIX: removed AND unit_price <= 999.99
    -- This filter was added March 2026 to mirror the GX expectation bound.
    -- It silently excluded all Bulk Commercial Supplies products (unit_price > 999.99)
    -- from August 28 through October 10, 2026 — 42 consecutive dbt runs.
    -- The correct approach: the GX checkpoint halts the ETL pipeline on failure,
    -- so dim_products only ever receives data that has passed all GX expectations.
    -- A static SQL price filter in dim_products is redundant when the upstream
    -- checkpoint enforces the price bound AND halts the pipeline on failure.
    AND unit_price > 0

Why GX checkpoint result isolation is a recurring Great Expectations retainer pattern

The Pinehurst Data Commerce incident demonstrates a failure class that is structural to how Great Expectations integrates with downstream pipeline steps. GX checkpoints are validation tools — they inspect data and report on quality. They do not automatically halt pipelines. The Python API returns a CheckpointResult object with a success attribute; whether the pipeline stops or continues is entirely up to the calling code. A data engineering team that adopts GX for observability — to surface data quality issues in Data Docs and Slack — without wiring the checkpoint result to pipeline control flow gets a system that correctly detects data quality problems and then loads the problematic data anyway.

The pattern is especially pernicious when combined with a downstream query or model filter that was written to match the expectation bounds. A dbt WHERE unit_price <= 999.99 filter and a GX expect_column_values_to_be_between expectation with max_value=999.99 encode the same business assumption in two different places: the GX expectation will alert when the assumption is violated; the dbt filter will silently enforce it. From the data warehouse’s perspective, the out-of-range rows never existed. No error appears in any query. No metric returns an unexpected value. The absence is only detectable when a stakeholder specifically looks for data they know should be there and finds it missing.

A Great Expectations developer retainer who performs a quarterly checkpoint-to-pipeline integration audit — verifying that every GX checkpoint result is checked with if not result.success: raise before the downstream load step; verifying that SlackNotificationAction channels have active subscribers with escalation paths; auditing dbt models for static filter conditions that duplicate GX expectation bounds (and therefore silently exclude data that GX flags as out-of-range rather than blocking it) — prevents this class of failure before 42 days of a product category being invisible to revenue attribution.

How HourTab helps Great Expectations developer retainers

The Pinehurst Data Commerce engagement totaled 5.5 hours: investigation (1.5h — 0.5h GX Data Docs review: 42 consecutive FAILED checkpoint runs confirmed August 28–October 10, all with 1,243 unexpected unit_price values in expect_column_values_to_be_between; 0.25h ETL source code review: absence of result[‘success’] check in loaders/product_catalog_loader.py confirmed; 0.25h dbt dim_products SQL review: WHERE unit_price <= 999.99 filter confirmed; 0.25h Slack channel audit: 42 delivered alerts to #data-quality-logs confirmed; channel subscriber count 0 confirmed; 0.25h Redshift staging vs. dim_products row count discrepancy confirmed: 1,243 Bulk Supplies rows in staging.product_catalog, 0 rows in dim_products WHERE product_category = ‘Bulk Commercial Supplies’), fix implementation and deployment (2.5h — 0.5h ETL pipeline updated: result[‘success’] check added; ValueError raised on failure with expectation detail; 0.5h GX expectation suite updated: max_value raised from 999.99 to 9999.99; meta note added; manual checkpoint run on October 10 data confirmed PASSED with 0 unexpected values; 0.5h dbt dim_products WHERE unit_price <= 999.99 filter removed; 0.25h SlackNotificationAction channel updated to #data-quality-alerts with 4 active subscribers; 0.25h changes deployed; integration test with synthetic out-of-range unit_price row confirmed checkpoint FAILED → ETL halts → dbt dim_products not triggered; 0.5h 42-day backfill: dbt dim_products re-run for August 28–October 10 to include all Bulk Supplies products), validation (1.5h — 0.5h Redshift dim_products confirmed 1,243 Bulk Supplies products present after backfill; product_category = ‘Bulk Commercial Supplies’ row count per day confirmed against invoicing system; 0.5h revenue attribution re-run: $502,481 Bulk Supplies revenue confirmed present in Q3 attribution totals after backfill; 0.5h documentation of root cause for postmortem: checkpoint result not checked, expectation suite staleness, dbt filter duplicating GX bound).

In a HourTab retainer log, the incident session reads: “product_catalog_checkpoint FAILED investigation: GX 0.18.15; expect_column_values_to_be_between unit_price 0.01–999.99; Bulk Commercial Supplies 1,243 products unit_price $1,200–$4,500 added August 28 2026; 42 consecutive FAILED checkpoint runs August 28–October 10; SlackNotificationAction fires to #data-quality-logs 0 subscribers; ETL loaders/product_catalog_loader.py loads data regardless of result[‘success’]; dbt dim_products WHERE unit_price <= 999.99 excludes all 1,243 Bulk Supplies products from dim_products; $502,481 Bulk Supplies revenue absent from Q3 analytics; fix: result[‘success’] check + ValueError raise on failure + max_value updated to 9999.99 + dbt filter removed + Slack channel updated to #data-quality-alerts 4 subscribers; 42-day backfill; 5.5h.”

HourTab turns a time-tracker CSV into a public retainer-hours URL your client can bookmark — no login, no portal, no “how many hours do I have left?” emails. Free for one retainer. $9/mo for up to ten.

FAQ: Great Expectations developer retainers

See also: dbt developer retainer · Apache Airflow developer retainer · Databricks developer retainer