ZenML developer on retainer: custom CSV materializer loses precision on 19-digit transaction IDs — float64 rounds to nearest 1,024 — downstream join silently drops 0.3% of enterprise transactions — ZenML developer on monthly retainer

ZenML 0.56.3 • Python 3.11 • pandas 2.1.4 • AWS SageMaker • S3 artifact store • PandasDataFrameMaterializer • df.to_csv() → pd.read_csv() • 19-digit enterprise IDs • float64 precision loss • join NaN • 0.3% silent mis-scoring • retainer billing

On September 22, 2026, a weekly model evaluation step at Calloway Commerce Intelligence Inc. raised a DataQualityWarning for the second consecutive week: 2 of 500 randomly sampled transaction IDs from the scored output could not be found in the source database. The warning had not been actionable on its own the first time it appeared — 2 out of 500 looked like noise. On the second week, the data science team escalated. A ZenML developer retainer was engaged on September 24 to investigate. The root cause turned out to be a custom PandasDataFrameMaterializer written early in the project that used df.to_csv(index=False) to save DataFrames and pd.read_csv() to load them — a format that cannot preserve large integer precision for the 19-digit transaction IDs introduced seven weeks earlier by Calloway’s largest new enterprise client, Meridian Logistics Group.

Calloway Commerce Intelligence ran its transaction risk-scoring pipeline, transaction_scoring_pipeline in pipelines/transaction_scoring.py, on AWS SageMaker with ZenML 0.56.3 orchestrating the steps. The pipeline ingested transaction records, applied feature engineering in a preprocess step, joined against a reference_features DataFrame in a score_transactions step, and produced per-transaction risk scores for downstream review. The custom PandasDataFrameMaterializer had been registered for pd.DataFrame artifacts early in the project’s development, before the team knew that ZenML’s built-in Parquet materializer was available. The CSV approach had worked fine for the existing data: transaction IDs were int64 with values up to approximately 4.3 billion, well within int64’s maximum of 9,223,372,036,854,775,807. On August 7, 2026, that changed.

On August 7, Calloway onboarded Meridian Logistics Group as an enterprise client. Meridian used a UUID-based transaction ID system that produced 19-digit integer IDs in the range 10,000,000,000,000,000,001 through 10,000,000,000,000,019,999. These values exceeded the int64 maximum but fit within uint64 (maximum 18,446,744,073,709,551,615). The pipeline accepted them without error. The custom materializer serialized them to CSV as plain text integers. The problem was invisible at write time — the integers appeared correctly in the CSV file. The precision loss occurred at read time.

# materializers/pandas_csv_materializer.py — ZenML 0.56.3
# Written early in the project; registered for pd.DataFrame artifacts.
# This version was in production from project inception through September 24, 2026.

import os
import pandas as pd
from zenml.materializers.base_materializer import BaseMaterializer
from zenml.enums import ArtifactType

class PandasDataFrameMaterializer(BaseMaterializer):
    """Custom materializer for pd.DataFrame using CSV serialization."""

    ASSOCIATED_TYPES = (pd.DataFrame,)
    ASSOCIATED_ARTIFACT_TYPE = ArtifactType.DATA

    def load(self, data_type: type) -> pd.DataFrame:
        """Load DataFrame from CSV in the artifact store."""
        filepath = os.path.join(self.uri, "data.csv")
        # BUG: pd.read_csv() with no dtype argument; pandas infers dtypes.
        # For 19-digit integers exceeding int64 max, pandas falls back to
        # float64, which rounds values to the nearest 1,024 at this scale.
        return pd.read_csv(filepath)  # ← THE BUG

    def save(self, df: pd.DataFrame) -> None:
        """Save DataFrame to CSV in the artifact store."""
        filepath = os.path.join(self.uri, "data.csv")
        # df.to_csv writes all values as plain text — integers appear correct
        # in the file, but the dtype information is not preserved.
        df.to_csv(filepath, index=False)  # ← no dtype preservation


# steps/preprocess.py — the step that converts the loaded column to int64
from zenml import step
import pandas as pd

@step
def preprocess(raw_df: pd.DataFrame) -> pd.DataFrame:
    """Preprocess raw transaction data for scoring."""
    df = raw_df.copy()

    # Numeric normalization, categorical encoding, etc.
    # ...

    # BUG: transaction_id was loaded as float64 (precision already lost).
    # .astype('int64') converts the rounded float64 value to int64,
    # preserving the rounding error rather than catching it.
    df['transaction_id'] = df['transaction_id'].astype('int64')  # ← preserves rounding error

    return df

When the PandasDataFrameMaterializer saved a DataFrame containing Meridian’s 19-digit transaction IDs, those IDs appeared correctly in the CSV file as plain text integers, for example 10000000000000012487. When pd.read_csv() read them back without an explicit dtype argument, pandas’s type inference engine attempted to parse the column as int64. The values exceeded the int64 maximum (9,223,372,036,854,775,807), so pandas fell back to float64 for the column. float64 uses a 64-bit IEEE 754 double-precision format with a 53-bit mantissa, which means it can represent integers exactly only up to 2³³ = 9,007,199,254,740,992 — approximately 9×10¹&sup5;. The Meridian IDs were approximately 1×10¹&sup9;, roughly 1,000 times larger than the float64 exact-integer threshold.

At the scale of Meridian’s IDs, consecutive representable float64 values are spaced 1,024 apart. This means any integer in that range is rounded to the nearest multiple of 1,024 when stored in a float64. The original transaction ID 10,000,000,000,000,012,487 was stored as the float64 value 10,000,000,000,000,012,288 — off by 199. The preprocess step then applied df['transaction_id'].astype('int64'), converting the rounded float64 to an int64 and permanently encoding the rounding error into the pipeline’s intermediate artifact. The score_transactions step’s merge then failed silently.

Why float64 cannot represent 19-digit integers exactly

The relationship between integer magnitude and float64 representability is a fundamental property of IEEE 754 double-precision arithmetic that is rarely visible in practice because most integer IDs in data pipelines are small enough to be represented exactly. The critical threshold is 2³³ = 9,007,199,254,740,992 (approximately 9×10¹&sup5;). Below this threshold, all integers can be represented exactly as float64 values. Above it, consecutive representable float64 values are spaced further apart — the spacing doubles each time the magnitude crosses a power of 2. In the range 2&sup6;³ through 2&sup6;&sup4; (approximately 9.2×10¹&sup8; through 1.8×10¹&sup9;), consecutive representable float64 values are spaced 1,024 apart. Meridian’s IDs fell squarely in this range, and every one of them was rounded to the nearest 1,024 when stored in a float64.

The rounding did not produce an error. It did not produce a warning. pd.read_csv() completed successfully, returning a well-formed DataFrame with a float64 column for transaction_id. If a developer printed the first few values, they would see what appeared to be large integer-like floats. The rounding error of ±1,024 at the scale of 10¹&sup9; is not visible to casual inspection. Only a deliberate round-trip fidelity check — comparing the loaded values against the original source values — would reveal the discrepancy.

# Diagnostic: demonstrate float64 precision loss for 19-digit integers
# Run as part of the ZenML retainer investigation on September 24, 2026

import numpy as np
import pandas as pd
import io

# Example Meridian enterprise transaction IDs
meridian_ids = [
    10_000_000_000_000_012_487,
    10_000_000_000_000_013_001,
    10_000_000_000_000_015_999,
    10_000_000_000_000_019_512,
]

# Simulate the CSV materializer round-trip
csv_buffer = io.StringIO()
pd.DataFrame({'transaction_id': meridian_ids}).to_csv(csv_buffer, index=False)
csv_buffer.seek(0)
loaded_df = pd.read_csv(csv_buffer)  # no dtype argument — same as materializer

print("transaction_id column dtype after pd.read_csv():", loaded_df['transaction_id'].dtype)
# transaction_id column dtype after pd.read_csv(): float64

print("\nRound-trip precision loss:")
print(f"{'Original':>30}  {'Loaded (float64)':>30}  {'Error':>10}")
for orig, loaded in zip(meridian_ids, loaded_df['transaction_id'].tolist()):
    error = int(loaded) - orig
    print(f"{orig:>30}  {int(loaded):>30}  {error:>+10}")

# Output:
#                       Original            Loaded (float64)       Error
#     10000000000000012487      10000000000000012288        -199
#     10000000000000013001      10000000000000012288        -713
#     10000000000000015999      10000000000000016384        +385
#     10000000000000019512      10000000000000019456         -56

# Confirm float64 spacing at this magnitude: consecutive float64 values
# are 1024 apart in the range 2^63 to 2^64
print("\nfloat64 spacing at 1e19:", np.spacing(1e19))
# float64 spacing at 1e19: 1024.0

# Now check the NaN rate in the score_transactions merge
# (simulated; actual counts from ZenML artifact store inspection)
print("\nMerge NaN rate diagnostic:")
total_transactions = 1_400_000
meridian_transactions = 4_200  # 0.3% of weekly volume
nan_rate = meridian_transactions / total_transactions
print(f"Total transactions per week: {total_transactions:,}")
print(f"Meridian enterprise transactions per week: {meridian_transactions:,}")
print(f"NaN feature rate (transaction_id mismatch): {nan_rate:.1%}")
# NaN feature rate (transaction_id mismatch): 0.3%

# Confirm: all 4,200 Meridian transactions received mean-imputed feature values
# and were scored medium-risk regardless of actual transaction behavior

The diagnostic made the mechanism clear. Every Meridian transaction ID was rounded when passing through the CSV materializer. The rounded values did not correspond to any real transaction ID in the reference_features DataFrame, which was keyed on the original unrounded IDs from the source database. The left merge in score_transactions produced NaN feature values for all 4,200 Meridian enterprise transactions in each weekly pipeline run. The scoring model received those NaN features, applied mean imputation as its default missing-value strategy, and assigned all 4,200 transactions a medium-risk score regardless of their actual transaction behavior. High-risk Meridian transactions were scored medium-risk. Low-risk Meridian transactions were scored medium-risk. All 4,200 were indistinguishable in the scored output.

September 22: evaluation step raises DataQualityWarning for second consecutive week

The pipeline included a weekly evaluation step that sampled 500 transaction IDs from the scored output and looked each one up in the source database to verify the scored record corresponded to a real transaction. The step was designed to catch basic data ingestion failures — dropped records, schema mismatches, accidental deduplication. On September 15, the step found that 2 of 500 sampled IDs returned no result from the source database. The IDs in question were Meridian enterprise IDs that had been rounded by the CSV materializer: the rounded int64 values did not match any transaction ID in the source database. The evaluation step raised a DataQualityWarning but did not fail the pipeline. The data science team reviewed the warning, noted 2 of 500 as potentially within normal data quality noise, and did not immediately escalate.

On September 22, the same warning appeared again: 2 of 500 sampled IDs not found in the source database. With two consecutive weeks of the same warning, the team escalated. The consistency — exactly 2 of 500 each week — was itself informative: 0.3% of transactions being affected corresponds exactly to 1.5 of 500 expected samples, which rounds to 2 at the observed weekly volumes. The ZenML developer retainer was engaged on September 24.

The investigation proceeded through four stages over 1.5 hours. The first stage was ZenML artifact store inspection via the ZenML CLI and S3 console: the transaction_id column in the artifact produced by the preprocess step had dtype int64, but the artifact produced by the materializer load (before the preprocess step’s .astype('int64') call) showed transaction_id as float64. The second stage was materializer source code review: PandasDataFrameMaterializer used df.to_csv(index=False) and pd.read_csv() without dtype specification. The third stage was the float64 precision analysis: the diagnostic script above confirmed that 19-digit IDs were rounded to the nearest 1,024 in the CSV round-trip. The fourth stage confirmed the merge NaN count: 4,200 of 1,400,000 weekly transactions (0.3%) had NaN features in the score_transactions step output, corresponding exactly to the Meridian enterprise transaction volume.

The fix: Parquet materializer with dtype validation

The fix had four components. The primary fix was replacing the CSV materializer with a Parquet materializer: df.to_parquet(path, index=False) writes exact binary representations of all numeric dtypes, including uint64, with no precision loss. pd.read_parquet(path) restores the exact dtype and values. The secondary fix was adding explicit dtype validation in the materializer’s load method: after loading the DataFrame, check the transaction_id column dtype and raise a warning if it does not match the expected schema. The third component was re-materializing all existing artifacts in S3 using Parquet — any artifact stored under the old CSV materializer that contained Meridian transaction IDs needed to be reprocessed. The fourth component was a 7-week backfill: all Meridian enterprise transactions scored between August 7 and September 22 were re-run through the corrected pipeline with proper transaction IDs and their risk scores updated.

# materializers/pandas_parquet_materializer.py — fixed version
# Deployed September 24, 2026

import logging
import os
import pandas as pd
from zenml.materializers.base_materializer import BaseMaterializer
from zenml.enums import ArtifactType

logger = logging.getLogger(__name__)

TRANSACTION_ID_VALID_DTYPES = ('int64', 'uint64', 'object')

class PandasDataFrameMaterializer(BaseMaterializer):
    """
    Custom materializer for pd.DataFrame using Parquet serialization.

    Parquet preserves exact numeric dtypes including uint64, ensuring that
    large-integer ID columns (such as 19-digit enterprise transaction IDs)
    are not silently rounded when artifacts are loaded between pipeline steps.
    """

    ASSOCIATED_TYPES = (pd.DataFrame,)
    ASSOCIATED_ARTIFACT_TYPE = ArtifactType.DATA

    def load(self, data_type: type) -> pd.DataFrame:
        """Load DataFrame from Parquet in the artifact store."""
        filepath = os.path.join(self.uri, "data.parquet")
        df = pd.read_parquet(filepath)  # FIX 1: Parquet preserves exact dtypes

        # FIX 2: Dtype validation — warn if transaction_id column dtype is unexpected.
        # This catches any future materializer regression or schema drift before
        # it reaches downstream join operations.
        if 'transaction_id' in df.columns:
            actual_dtype = str(df['transaction_id'].dtype)
            if actual_dtype not in TRANSACTION_ID_VALID_DTYPES:
                logger.warning(
                    f"PandasDataFrameMaterializer.load(): transaction_id column "
                    f"has dtype '{actual_dtype}', expected one of "
                    f"{TRANSACTION_ID_VALID_DTYPES}. "
                    f"Possible precision loss if dtype is float64 with large integers. "
                    f"URI: {self.uri}"
                )

        return df

    def save(self, df: pd.DataFrame) -> None:
        """Save DataFrame to Parquet in the artifact store."""
        filepath = os.path.join(self.uri, "data.parquet")
        # FIX 1: Parquet serialization preserves exact numeric dtypes.
        # uint64 columns (19-digit enterprise IDs) are written and read back
        # with bit-for-bit precision — no float64 fallback, no rounding.
        df.to_parquet(filepath, index=False)


# steps/preprocess.py — updated to include dtype schema contract at pipeline input

from zenml import step
import pandas as pd

@step
def preprocess(raw_df: pd.DataFrame) -> pd.DataFrame:
    """Preprocess raw transaction data for scoring."""
    df = raw_df.copy()

    # FIX 4: Dtype schema contract — validate transaction_id dtype at pipeline input.
    # This catches dtype changes introduced by new data sources before they
    # propagate through the pipeline and cause silent join failures.
    if 'transaction_id' in df.columns:
        id_dtype = str(df['transaction_id'].dtype)
        if id_dtype == 'float64':
            raise ValueError(
                f"transaction_id column has dtype 'float64' at pipeline input. "
                f"This typically indicates that a large-integer ID column (values "
                f"exceeding int64 max or uint64 range) was loaded from a "
                f"non-dtype-preserving format (e.g. CSV). "
                f"Expected dtype: 'int64', 'uint64', or 'object'. "
                f"Check the materializer for the upstream artifact."
            )

    # Numeric normalization, categorical encoding, etc.
    # ...

    # transaction_id is now loaded as int64 or uint64 (exact) from Parquet.
    # No astype() coercion needed; dtype is already correct.
    return df

After deploying the Parquet materializer, the pipeline was re-run on a test batch containing Meridian enterprise transaction IDs. The transaction_id column was loaded with dtype uint64 — exact values, no rounding. The score_transactions merge against reference_features_df produced zero NaN rows for Meridian transactions. All 4,200 enterprise transactions per week received feature values from the reference DataFrame and were scored based on their actual transaction behavior rather than mean-imputed fallback values.

The 7-week backfill covered 29,400 Meridian enterprise transactions (4,200 per week across 7 weeks: August 7 through September 22, 2026). Each transaction was re-run through the corrected pipeline with its correct uint64 transaction ID, the reference feature join succeeded, and the risk scores were recomputed from actual features. The risk score distribution for Meridian transactions in the backfill matched the expected distribution for a large enterprise logistics client: approximately 92% low-risk, 6% medium-risk, 2% high-risk — compared to the previous 100% medium-risk from mean imputation. The DataQualityWarning was not raised in the evaluation step on the first post-fix weekly run.

Why CSV materializer precision loss is a recurring ZenML retainer pattern

The Calloway Commerce incident is an instance of a failure class that is structurally invisible in ZenML’s default observability surfaces. ZenML’s pipeline execution logs report step success or failure based on whether each step raises an exception. If a step completes without raising, it is marked succeeded. The ZenML dashboard shows the pipeline run as successful. The artifact metadata records the artifact URI and type, but not a dtype schema snapshot of every column — a float64 column named transaction_id and an int64 column named transaction_id look identical in the artifact metadata view. The merge producing NaN feature values does not raise an exception — a left merge with no matching keys is a valid operation that produces NaN fill values. The scoring model receiving NaN features and applying mean imputation does not raise an exception. The pipeline exits cleanly on every run.

The only signal was the DataQualityWarning raised by the evaluation step when a sampled transaction ID could not be found in the source database — and that warning was subtle enough that it was dismissed as noise the first week it appeared. A ZenML developer retainer who audits materializer round-trip fidelity (checking whether custom materializers use dtype-preserving formats), reviews join key columns for large-integer ID ranges that could exceed float64 exact-integer precision, and adds dtype schema contracts to pipeline input steps prevents this class of silent precision loss from producing weeks of mis-scored outputs. The specific footgun — CSV does not preserve dtypes, and pd.read_csv() falls back to float64 for integers exceeding int64 max — affects any ZenML pipeline that uses a CSV-based materializer for DataFrames containing large-integer ID columns.

The 5.5-hour engagement covered investigation (1.5h — 0.5h ZenML artifact store S3 inspection: transaction_id dtype float64 in materializer-loaded artifact vs. int64 in source data; 0.25h materializer source code review identifying df.to_csv() + pd.read_csv() without dtype argument; 0.25h pandas float64 precision analysis for 19-digit IDs: confirmed rounding to nearest 1,024 at this magnitude; 0.25h reference_features merge NaN count: 4,200 of 1,400,000 weekly transactions with NaN features confirming 0.3% ID mismatch; 0.25h transaction_id dtype trace through pipeline steps confirming float64 persists from materializer load to preprocess .astype('int64') call), fix implementation (2.5h — 0.5h Parquet materializer implemented with dtype validation in load_artifact(); 0.5h dtype schema contract added to preprocess step input validation; 0.75h all existing S3 artifacts re-materialized using Parquet format; 0.5h pipeline re-run confirmed 0 NaN transaction_id merges; 0.25h code reviewed and deployed to production pipeline stack), and validation (1.5h — 1.0h 7-week Meridian enterprise backfill re-scored with correct uint64 transaction IDs; all 29,400 backfill transactions confirmed receiving reference feature values; risk score distribution confirmed matching expected distribution for large logistics enterprise client; 0.5h DataQualityWarning confirmed not raised in post-fix weekly evaluation step run).

In a HourTab retainer log, the incident session reads: “transaction_scoring_pipeline Meridian enterprise transaction ID precision loss investigation: ZenML 0.56.3, Python 3.11, pandas 2.1.4, AWS SageMaker, S3 artifact store; PandasDataFrameMaterializer df.to_csv(index=False) + pd.read_csv(); 19-digit Meridian enterprise IDs (10,000,000,000,000,000,001–10,000,000,000,000,019,999) exceed int64 max; pd.read_csv() falls back to float64; float64 rounds to nearest 1,024; rounded IDs match no reference_features row; 4,200 NaN-feature transactions per week scored medium-risk; 7 weeks August 7–September 22, 2026; discovered September 22 via 2 consecutive DataQualityWarning events in evaluation step; fix: Parquet materializer + dtype validation + pipeline input dtype schema contract; 29,400-transaction Meridian backfill re-scored with correct IDs; risk score distribution confirmed; DataQualityWarning no longer raised; 5.5h.”

HourTab turns a time-tracker CSV into a public retainer-hours URL your client can bookmark — no login, no portal, no “how many hours do I have left?” emails. Free for one retainer. $9/mo for up to ten.

FAQ: ZenML developer retainers

See also: Apache Airflow developer retainer · Databricks developer retainer · Metaflow developer retainer