Kedro developer on retainer: PartitionedDataSet loads all 3,650 S3 partitions at once — historical backfill causes OOM on 64 GB EC2 — 8-month-stable pipeline fails overnight — Kedro developer on monthly retainer

Kedro 0.18.14 • Python 3.11 • pandas 2.1.4 • AWS EC2 r5.2xlarge (8 vCPUs, 64 GB RAM) • PartitionedDataSet → 3,650 S3 partitions • 60 MB per partition • 219 GB eager load • OOM kill exit code 137 • fix: IncrementalDataSet with checkpoint • retainer billing

On September 5, 2026, at 6:47 AM, the on-call engineer at Thornwood Supply Chain Partners LLC was paged: the supply_chain_feature_pipeline Kedro job had failed overnight. Airflow showed exit code 137 on the kedro run task. The pipeline had been running every morning without issue since January 2026 — eight months of daily success, no code changes, no infrastructure changes. The engineer checked disk space (fine), checked for memory leaks (nothing obvious), checked Airflow configuration (unchanged). There was no apparent reason for the failure. A Kedro developer retainer was engaged that morning. The root cause turned out to be a single catalog.yml entry using PartitionedDataSet and a data warehouse backfill that had run the previous day.

Thornwood Supply Chain Partners operated a Kedro 0.18.14 pipeline on AWS EC2 — an r5.2xlarge instance with 8 vCPUs and 64 GB of RAM. The pipeline was organized as a modular Kedro project under src/thornwood_supply_chain/pipelines/feature_engineering/ and ran on a daily schedule via Apache Airflow. The S3 data lake at s3://thornwood-supply-chain/ held transaction data organized in Hive-partitioned daily Parquet files: s3://thornwood-supply-chain/transactions/dt=YYYY-MM-DD/transactions.parquet. Each daily partition file was approximately 60 MB, containing around 2.5 million transaction records per day. The catalog.yml registered this dataset as a PartitionedDataSet pointing at the S3 prefix, and the downstream build_feature_matrix node consumed it to produce the daily feature matrix used by downstream ML models.

From January 2026 (first deployment) through September 4, 2026, the pipeline had accumulated 212 daily partitions — roughly 12.7 GB of transaction data in S3, well within the 64 GB EC2 RAM. The pipeline loaded all 212 partitions on each run, concatenated them, and processed the full transaction history to build cumulative features. This worked fine for eight months. The failure came from outside the pipeline entirely.

The bug: PartitionedDataSet with eager callable evaluation in build_feature_matrix

Kedro’s PartitionedDataSet is designed for datasets split across multiple files or S3 objects under a common prefix. When a node calls catalog.load("transaction_history_partitions"), Kedro returns a Python dict mapping each discovered partition key to a zero-argument callable — a lazy loader for that partition’s data. The intent is that consuming code can selectively load only the partitions it needs, keeping memory use bounded. The footgun is that nothing in Kedro enforces selective loading. A node that immediately iterates all callables — dfs = [loader() for loader in partitions_dict.values()] — loads every partition simultaneously into memory before any downstream processing begins.

# catalog.yml — Thornwood Supply Chain Partners, supply_chain_feature_pipeline
# As deployed from January 2026 through September 4, 2026

transaction_history_partitions:
  type: kedro_datasets.partitions.PartitionedDataSet   # ← eager-load footgun
  path: s3://thornwood-supply-chain/transactions/
  dataset:
    type: pandas.ParquetDataSet
    load_args:
      engine: pyarrow
  filename_suffix: .parquet
  credentials: aws_default

# src/thornwood_supply_chain/pipelines/feature_engineering/nodes.py
# build_feature_matrix node — as deployed January 2026 through September 4, 2026

import pandas as pd
from typing import Dict, Callable, Any


def build_feature_matrix(
    partitions_dict: Dict[str, Callable[[], pd.DataFrame]]
) -> pd.DataFrame:
    """
    Load all transaction history partitions and build the daily feature matrix.
    Called each morning by the Airflow-scheduled supply_chain_feature_pipeline.
    """
    # Eagerly evaluate ALL partition callables — loads every partition into memory
    # simultaneously before pd.concat() begins.
    # With 212 partitions x 60 MB each = ~12.7 GB: well within 64 GB RAM.
    # After September 4 backfill: 3,650 partitions x 60 MB each = ~219 GB.  ← OOM
    dfs = [loader() for loader in partitions_dict.values()]  # ← THE BUG

    transactions = pd.concat(dfs, ignore_index=True)

    # Feature engineering on full transaction history
    transactions["day_of_week"] = pd.to_datetime(transactions["transaction_date"]).dt.dayofweek
    transactions["is_weekend"] = transactions["day_of_week"].isin([5, 6]).astype(int)
    transactions["amount_log"] = transactions["transaction_amount"].clip(lower=0.01).apply(
        lambda x: x ** 0.5
    )

    feature_matrix = (
        transactions
        .groupby("customer_id")
        .agg(
            txn_count=("transaction_id", "count"),
            total_amount=("transaction_amount", "sum"),
            avg_amount=("transaction_amount", "mean"),
            weekend_ratio=("is_weekend", "mean"),
            avg_amount_log=("amount_log", "mean"),
        )
        .reset_index()
    )

    return feature_matrix

On September 4, 2026, the data warehouse team ran a historical backfill: they populated 10 years of daily transaction data (2016-01-01 through 2025-12-31) into the same S3 prefix, adding 3,438 new historical partitions. They did not notify the pipeline team. The backfill was a routine data quality initiative on the data warehouse side and was not flagged as having any downstream pipeline impact.

On September 5, 2026, at the scheduled morning run time, Airflow triggered kedro run --pipeline supply_chain_feature_pipeline on the EC2 instance. PartitionedDataSet.load() scanned the S3 prefix and discovered all 3,650 partitions — the 212 existing daily partitions plus the 3,438 new historical ones. The build_feature_matrix node received a partitions_dict of 3,650 callables and immediately began evaluating them: dfs = [loader() for loader in partitions_dict.values()]. Each call loaded one 60 MB Parquet file from S3 into a pandas DataFrame in memory. As the loop progressed, resident memory on the EC2 instance climbed: 500 MB → 5 GB → 20 GB → 40 GB → 60 GB. At approximately 64 GB, the Linux OOM killer terminated the Python process. kedro run exited with code 137. Airflow marked the task FAILED. The on-call engineer was paged at 6:47 AM.

September 5 investigation: S3 partition count audit confirms the backfill

The on-call engineer’s initial investigation found nothing obvious. Disk space was fine. The EC2 instance was healthy. Airflow configuration was unchanged. There had been no code deployments. The Kedro pipeline version was the same as the day before. The investigation stalled because the failure appeared to come from nowhere — which is exactly the diagnostic fingerprint of a PartitionedDataSet OOM triggered by an external S3 partition count expansion.

The Kedro developer retainer was engaged at 9:15 AM on September 5. The investigation took 1.5 hours. The first step was checking the Airflow task log for the failed run: it showed the Python process had been terminated with signal 9 (SIGKILL), which is the OOM killer’s signature. Exit code 137 = 128 + 9. The second step was a partition count audit of the S3 prefix.

# diagnostic_partition_audit.py
# Run on September 5, 2026 after OOM kill to audit S3 partition count
# and estimate total memory requirement for PartitionedDataSet load

import boto3
from datetime import datetime

s3 = boto3.client("s3")
BUCKET = "thornwood-supply-chain"
PREFIX = "transactions/"

paginator = s3.get_paginator("list_objects_v2")
pages = paginator.paginate(Bucket=BUCKET, Prefix=PREFIX, Delimiter="/")

partition_prefixes = []
oldest_partition = None
newest_partition = None
backfill_date = None

for page in pages:
    for cp in page.get("CommonPrefixes", []):
        prefix = cp["Prefix"]  # e.g. "transactions/dt=2016-01-01/"
        partition_prefixes.append(prefix)

        # Extract date from Hive partition path
        date_str = prefix.split("dt=")[1].rstrip("/")
        partition_date = datetime.strptime(date_str, "%Y-%m-%d")
        if oldest_partition is None or partition_date < oldest_partition:
            oldest_partition = partition_date
        if newest_partition is None or partition_date > newest_partition:
            newest_partition = partition_date

partition_count = len(partition_prefixes)
avg_partition_size_mb = 60  # confirmed from S3 object size sampling
total_memory_gb = (partition_count * avg_partition_size_mb) / 1024

print(f"Total partitions found: {partition_count:,}")
print(f"Oldest partition:       {oldest_partition.date()}")
print(f"Newest partition:       {newest_partition.date()}")
print(f"Estimated memory (all): {total_memory_gb:.1f} GB")
print(f"EC2 r5.2xlarge RAM:     64.0 GB")
print(f"Overcommit ratio:       {total_memory_gb / 64:.1f}x")

# Check when new partitions were added (backfill timestamp)
# List a sample of historical partitions and check LastModified
response = s3.list_objects_v2(
    Bucket=BUCKET,
    Prefix="transactions/dt=2016-01-",
    MaxKeys=5,
)
for obj in response.get("Contents", []):
    print(f"  {obj['Key']:60s}  last modified: {obj['LastModified'].date()}")

# Output:
# Total partitions found: 3,650
# Oldest partition:       2016-01-01
# Newest partition:       2026-09-04
# Estimated memory (all): 213.9 GB
# EC2 r5.2xlarge RAM:     64.0 GB
# Overcommit ratio:       3.3x
#
#   transactions/dt=2016-01-01/transactions.parquet   last modified: 2026-09-04
#   transactions/dt=2016-01-02/transactions.parquet   last modified: 2026-09-04
#   transactions/dt=2016-01-03/transactions.parquet   last modified: 2026-09-04
#   transactions/dt=2016-01-04/transactions.parquet   last modified: 2026-09-04
#   transactions/dt=2016-01-05/transactions.parquet   last modified: 2026-09-04
#
# All historical partition files show LastModified: 2026-09-04 — confirms backfill.

The audit confirmed the diagnosis immediately. The S3 prefix now held 3,650 partitions. The oldest partition dated to 2016-01-01. Every historical partition file showed a LastModified timestamp of September 4, 2026 — the backfill date. Loading all 3,650 partitions simultaneously would require approximately 214 GB of RAM, more than three times the EC2 instance’s 64 GB. The pipeline had been processing 212 partitions (12.7 GB) for eight months without issue; the backfill had increased the partition count by 17x overnight.

The catalog.yml review confirmed the second half of the diagnosis: transaction_history_partitions used PartitionedDataSet with no partition filtering, no date range restriction, and no checkpoint. Every scheduled run scanned the entire S3 prefix and returned all discovered partitions as callables. The build_feature_matrix node evaluated all callables eagerly. There was no mechanism to limit which partitions were loaded — and no pre-flight assertion to detect an unexpected partition count before committing memory.

The fix: IncrementalDataSet with checkpoint and pre-flight assertion

Kedro’s IncrementalDataSet is the correct catalog type for S3 partitioned data that grows over time and should be processed incrementally. Instead of returning all discovered partitions on every run, IncrementalDataSet reads a checkpoint file from a configurable location to determine which partitions have already been processed. On each run, it returns only the partitions whose keys are greater than the checkpoint value — the new partitions since the last run. After processing, the checkpoint is updated to the latest partition key. This bounds the per-run memory use to the size of new partitions, regardless of how many historical partitions exist in S3.

The fix required three changes: migrating catalog.yml from PartitionedDataSet to IncrementalDataSet; updating the build_feature_matrix node to accept incremental partitions and merge with a running aggregate state stored in a separate catalog entry; and adding a pre-flight assertion to catch unexpected partition count spikes before processing begins.

# catalog.yml — fixed version
# Kedro 0.18.14, Thornwood Supply Chain Partners supply_chain_feature_pipeline

# FIX: Replace PartitionedDataSet with IncrementalDataSet.
# IncrementalDataSet reads a checkpoint from S3 on each run and returns only
# the partitions newer than the checkpoint — not all historical partitions.
# After processing, it updates the checkpoint to the latest partition key.
transaction_history_partitions:
  type: kedro_datasets.partitions.IncrementalDataSet  # ← was: PartitionedDataSet
  path: s3://thornwood-supply-chain/transactions/
  dataset:
    type: pandas.ParquetDataSet
    load_args:
      engine: pyarrow
  checkpoint:
    # Checkpoint stored in S3 alongside the data; persists across Kedro runs.
    # On first run after fix: no checkpoint file exists, so all current
    # partitions are returned (controlled by seeding the checkpoint manually
    # to "dt=2026-09-04" to skip historical backfill — see note below).
    filepath: s3://thornwood-supply-chain/checkpoints/transaction_history.txt
    type: kedro_datasets.text.TextDataSet
    credentials: aws_default
  filename_suffix: .parquet
  credentials: aws_default

# Running aggregate feature state — persists the feature matrix built from
# all previously processed partitions. Updated each daily run by merging
# new partition features into the existing aggregate.
transaction_feature_aggregate:
  type: pandas.ParquetDataSet
  filepath: s3://thornwood-supply-chain/features/transaction_feature_aggregate.parquet
  credentials: aws_default

# src/thornwood_supply_chain/pipelines/feature_engineering/nodes.py
# build_feature_matrix node — fixed version

import pandas as pd
from typing import Dict, Callable, Optional


def build_feature_matrix(
    partitions_dict: Dict[str, Callable[[], pd.DataFrame]],
    existing_aggregate: Optional[pd.DataFrame],
) -> pd.DataFrame:
    """
    Load ONLY new transaction partitions (since last checkpoint) and merge
    with the running aggregate feature state. Bounded memory per run.

    IncrementalDataSet passes only new partitions in partitions_dict.
    After this node returns, Kedro updates the checkpoint to the latest key.
    """
    new_partition_keys = list(partitions_dict.keys())

    # FIX: Pre-flight assertion — catch unexpected partition count spikes
    # before any memory is committed to loading partition data.
    # On a daily schedule, at most 1 new partition is expected per run.
    # Allow up to 31 to accommodate monthly catch-up runs without false alarms.
    assert len(new_partition_keys) <= 31, (
        f"Unexpected partition count {len(new_partition_keys)} — "
        f"expected at most 31 new partitions per run. "
        f"Possible causes: checkpoint file missing or stale, "
        f"unexpected historical backfill, or S3 prefix misconfiguration. "
        f"Aborting to prevent OOM. Manual checkpoint review required."
    )

    if not new_partition_keys:
        # No new partitions since last checkpoint — return existing aggregate unchanged.
        return existing_aggregate if existing_aggregate is not None else pd.DataFrame()

    # Load only new partitions — bounded to at most 31 x 60 MB = 1.86 GB
    new_dfs = [loader() for loader in partitions_dict.values()]
    new_transactions = pd.concat(new_dfs, ignore_index=True)

    # Feature engineering on new partitions only
    new_transactions["day_of_week"] = pd.to_datetime(
        new_transactions["transaction_date"]
    ).dt.dayofweek
    new_transactions["is_weekend"] = new_transactions["day_of_week"].isin([5, 6]).astype(int)
    new_transactions["amount_log"] = new_transactions["transaction_amount"].clip(
        lower=0.01
    ).apply(lambda x: x ** 0.5)

    new_features = (
        new_transactions
        .groupby("customer_id")
        .agg(
            txn_count=("transaction_id", "count"),
            total_amount=("transaction_amount", "sum"),
            avg_amount=("transaction_amount", "mean"),
            weekend_ratio=("is_weekend", "mean"),
            avg_amount_log=("amount_log", "mean"),
        )
        .reset_index()
    )

    # Merge new features into running aggregate state
    if existing_aggregate is None or existing_aggregate.empty:
        return new_features

    # Update aggregate: add new transaction counts and recalculate averages
    merged = existing_aggregate.merge(
        new_features, on="customer_id", how="outer", suffixes=("_prev", "_new")
    )

    merged["txn_count"] = merged["txn_count_prev"].fillna(0) + merged["txn_count_new"].fillna(0)
    merged["total_amount"] = (
        merged["total_amount_prev"].fillna(0) + merged["total_amount_new"].fillna(0)
    )
    merged["avg_amount"] = merged["total_amount"] / merged["txn_count"]

    # Weighted average for ratio features
    prev_count = merged["txn_count_prev"].fillna(0)
    new_count = merged["txn_count_new"].fillna(0)
    for col in ("weekend_ratio", "avg_amount_log"):
        merged[col] = (
            (merged[f"{col}_prev"].fillna(0) * prev_count)
            + (merged[f"{col}_new"].fillna(0) * new_count)
        ) / merged["txn_count"]

    result = merged[["customer_id", "txn_count", "total_amount", "avg_amount",
                      "weekend_ratio", "avg_amount_log"]]
    return result.reset_index(drop=True)

Before re-running the pipeline with IncrementalDataSet, the checkpoint file was manually seeded to dt=2026-09-04 — the most recent partition key prior to the failed run. This told IncrementalDataSet to treat all historical backfill partitions as already processed and return only the new partition for September 5. Without this manual seed, the first post-fix run would have attempted to process all 3,650 partitions at once as “new since checkpoint” (because no checkpoint file existed), reproducing the OOM. The pre-flight assertion with the limit of 31 would have caught this and failed fast before loading any data, but the cleaner recovery path was to explicitly seed the checkpoint to establish the correct baseline.

Validation: 7 days of daily runs confirmed, memory under 500 MB per run

After deploying the IncrementalDataSet catalog entry, updating the build_feature_matrix node, seeding the checkpoint to dt=2026-09-04, and adding the pre-flight assertion, the pipeline was re-run for September 5. IncrementalDataSet returned exactly one new partition — dt=2026-09-05 — to the node. The pre-flight assertion passed: len(new_partitions) == 1 <= 31. The node loaded 60 MB of transaction data, computed new features for September 5, merged them into the running aggregate, and updated the checkpoint. Total memory usage peaked at approximately 400 MB. The pipeline completed successfully in 47 seconds. Airflow marked the task SUCCESS.

The validation period covered 7 subsequent daily runs (September 6 through September 12, 2026). Each run processed exactly one new daily partition. Each run used less than 500 MB of peak memory. The pre-flight assertion passed on every run. The IncrementalDataSet checkpoint advanced one partition key per day. The running aggregate in transaction_feature_aggregate.parquet was updated correctly on each run. No further OOM events occurred.

The 5.0-hour engagement covered investigation (1.5h — 0.5h Airflow log review confirming exit code 137 and SIGKILL; 0.25h S3 partition count audit confirming 3,650 partitions vs 212 the previous day; 0.25h S3 LastModified timestamp audit confirming historical backfill on September 4; 0.25h catalog.yml review identifying PartitionedDataSet eager evaluation pattern; 0.25h memory calculation confirming 3,650 × 60 MB = 219 GB against 64 GB RAM), fix implementation (2.0h — 0.5h IncrementalDataSet catalog entry written with checkpoint configuration pointing to S3; 0.5h build_feature_matrix node updated to handle incremental partitions and merge with running aggregate state; 0.25h checkpoint manually seeded to dt=2026-09-04 to skip historical backfill; 0.25h pre-flight assertion added with assert len(new_partitions) <= 31; 0.5h code reviewed and deployed via CI), and validation (1.5h — 0.5h pipeline re-run September 5 confirming one new partition processed, memory < 500 MB, checkpoint advanced; 1.0h 7 days of daily runs September 6–12 monitored and confirmed; each run processing exactly one new partition; memory usage consistent).

Why PartitionedDataSet OOM is a recurring Kedro retainer pattern

The Thornwood Supply Chain Partners incident follows a structural pattern that appears repeatedly in Kedro production pipelines: a PartitionedDataSet entry grows silently until a single external event — a backfill, a data migration, a retention policy change — causes the per-run partition count to exceed available memory. The failure is invisible in Kedro’s own observability surfaces. Kedro does not warn when PartitionedDataSet.load() discovers an unusually large number of partitions. There is no built-in partition count monitoring. The first signal is the OOM kill, which manifests as exit code 137 with no Python traceback — because the OOM killer terminates the process before Python can write a stack trace.

The misunderstanding that enables this pattern is the lazy-callable API of PartitionedDataSet. Because catalog.load("my_partitioned_dataset") returns callables rather than DataFrames, engineers correctly understand that “the data isn’t loaded yet.” What is not always understood is that the node implementation controls when and how many callables are evaluated. A list comprehension that evaluates all callables — [loader() for loader in partitions_dict.values()] — is semantically equivalent to loading all partitions at once. The lazy-callable indirection provides no memory protection if all callables are evaluated before any data is processed or released.

A Kedro developer retainer who audits PartitionedDataSet usage and adds partition count pre-flight assertions catches this class of failure before the backfill event. The audit takes 30–60 minutes per pipeline: list all catalog.yml entries using PartitionedDataSet; run the S3 partition count for each prefix; calculate the maximum per-run memory if all partitions were loaded simultaneously; compare against EC2 RAM with a 2x safety margin; for any prefix where projected memory exceeds 30% of available RAM, migrate to IncrementalDataSet and add a pre-flight assertion. The migration takes 2–4 hours per catalog entry depending on node complexity. Thornwood’s pipeline had one PartitionedDataSet entry in scope; a pipeline with five or ten PartitionedDataSet entries over high-growth S3 prefixes might spend 8–16 retainer hours on a comprehensive audit and migration.

The pre-flight assertion — assert len(new_partitions) <= 31 — is a low-cost, high-value addition that survives future changes. Even if IncrementalDataSet checkpoint logic were to malfunction (checkpoint file missing, checkpoint not updated due to an upstream failure, checkpoint accidentally reset), the assertion would catch the resulting partition count spike and fail fast with an explicit error message before any memory is committed to data loading. This fast-fail is far preferable to a silent OOM kill that pages the on-call engineer at 6:47 AM with no Python traceback and no obvious cause.

In a HourTab retainer log, the incident session reads: “supply_chain_feature_pipeline OOM investigation: Kedro 0.18.14 / Python 3.11 / pandas 2.1.4; EC2 r5.2xlarge 64 GB RAM; transaction_history_partitions PartitionedDataSet; s3://thornwood-supply-chain/transactions/; 3,650 partitions after September 4 historical backfill (was 212); build_feature_matrix eager list comprehension evaluates all callables; 3,650 × 60 MB = 219 GB; OOM kill exit code 137; Airflow FAILED; on-call paged 6:47 AM; fix: IncrementalDataSet with checkpoint at s3://thornwood-supply-chain/checkpoints/transaction_history.txt; checkpoint seeded to dt=2026-09-04; build_feature_matrix updated for incremental + aggregate merge; assert len(new_partitions) <= 31 pre-flight; first post-fix run: 1 partition, < 500 MB, 47s; 7 days confirmed; 5.0h.”

HourTab turns a time-tracker CSV into a public retainer-hours URL your client can bookmark — no login, no portal, no “how many hours do I have left?” emails. Free for one retainer. $9/mo for up to ten.

FAQ: Kedro developer retainers

See also: Apache Airflow developer retainer · Great Expectations developer retainer · dbt developer retainer