Metaflow developer on retainer: self.model_artifacts pickle serialization raises ModuleNotFoundError in AWS Batch container — evaluate step fails every run — 3 months of manual artifact workaround

Metaflow 2.11.5 • AWS Batch • Python 3.11 • S3 artifact storage • inline FeatureEncoder class • pickle module path serialization • ModuleNotFoundError: No module named 'flows' • evaluate step fails every run • retainer billing

On October 13, 2026, a new ML engineer joined Summit Ridge ML Systems Inc. and spent their first afternoon reviewing the ModelTrainingFlow pipeline. Within an hour they sent a message to the #ml-platform Slack channel: “Why does the evaluate step always fail on first run and require manual intervention? Is that expected?” The question landed in a channel that had been treating this as normal for twelve weeks. The senior ML engineer replied: “Yeah it’s a known issue, I run a script to fix the artifact and re-trigger it manually after each training run.” When a Metaflow developer retainer was engaged the same day, the investigation uncovered a structural serialization incompatibility that had been failing every run since July 14, 2026 — and had been silently absorbed into the team’s weekly operations as a normalized manual process.

Summit Ridge ML Systems ran a Metaflow 2.11.5 pipeline on AWS for weekly model training and evaluation. The ModelTrainingFlow in flows/model_training_flow.py consisted of a start step, a training step decorated with @batch(cpu=4, memory=32000) that ran on AWS Batch using the summit-ridge-ml:latest container image, and an evaluate step that consumed the training step’s artifacts via self.model_artifacts. Metaflow persisted all self.* artifact assignments to S3 using Python’s pickle protocol, then deserialized them in the downstream step’s Batch container at runtime. The flow had been running weekly since April 2026 without issue — until a FeatureEncoder class was added to the training step on July 14, 2026.

# flows/model_training_flow.py — Metaflow 2.11.5
# As it existed from July 14, 2026 onward (the problematic version)

from metaflow import FlowSpec, step, batch, retry, S3
import numpy as np
import pandas as pd
import pickle


# FeatureEncoder defined inline in the flow file — not in an installed package
class FeatureEncoder:
    def __init__(self, categories):
        self.categories = categories
        self.encoder_map = {cat: i for i, cat in enumerate(categories)}

    def encode(self, value):
        return self.encoder_map.get(value, -1)


class ModelTrainingFlow(FlowSpec):

    @step
    def start(self):
        self.next(self.training)

    @batch(cpu=4, memory=32000, image="summit-ridge-ml:latest")
    @retry(times=2)
    @step
    def training(self):
        from sklearn.linear_model import LogisticRegression
        from sklearn.preprocessing import StandardScaler

        # Load training data from S3
        df = pd.read_parquet("s3://summit-ridge-data/features/latest.parquet")
        categories = df["product_category"].unique().tolist()

        # FeatureEncoder is defined in this file — inline in flows/model_training_flow.py
        encoder = FeatureEncoder(categories)
        X = np.array([encoder.encode(c) for c in df["product_category"]]).reshape(-1, 1)
        y = df["converted"].values

        scaler = StandardScaler()
        X_scaled = scaler.fit_transform(X)

        model = LogisticRegression()
        model.fit(X_scaled, y)

        # self.model_artifacts stores an instance of FeatureEncoder
        # pickle will serialize FeatureEncoder as: flows.model_training_flow.FeatureEncoder
        # This is the fully qualified module path — pickle does NOT embed the class definition
        self.model_artifacts = {
            "model": model,
            "encoder": encoder,          # ← pickle stores this as flows.model_training_flow.FeatureEncoder
            "scaler": scaler,
            "categories": categories,
        }
        self.next(self.evaluate)

    @batch(cpu=2, memory=8000, image="summit-ridge-ml:latest")
    @step
    def evaluate(self):
        # Metaflow deserializes self.model_artifacts from S3 here.
        # pickle.loads() tries to import flows.model_training_flow and access FeatureEncoder.
        # The evaluate step's Batch container was built from requirements.txt only —
        # the flows/ directory is NOT in the container's Python path.
        # Result: ModuleNotFoundError: No module named 'flows'

        artifacts = self.model_artifacts   # ← this line raises ModuleNotFoundError
        encoder = artifacts["encoder"]
        model = artifacts["model"]
        scaler = artifacts["scaler"]

        # ... evaluation logic using encoder, model, scaler
        self.next(self.end)

    @step
    def end(self):
        pass


if __name__ == "__main__":
    ModelTrainingFlow()

The FeatureEncoder class had been added on July 14, 2026 by the senior ML engineer to handle categorical feature encoding in a reproducible way during both training and serving. Defining it in the flow file seemed natural at the time — it was small, it was only used in the training flow, and there was no existing utility package to add it to. The engineer tested the training step in isolation and it worked. The evaluate step, however, was the first step in the pipeline that ran in a fresh AWS Batch container and attempted to deserialize self.model_artifacts from the S3 pickle artifact. That deserialization had been failing on every run since the class was introduced.

The Docker image summit-ridge-ml:latest was built from a requirements.txt that listed Metaflow, scikit-learn, pandas, numpy, and their transitive dependencies. The image was maintained by the infrastructure team and rebuilt whenever a new Python package was needed. Crucially, the image was not built from the source repository — the flows/ directory was never copied into the image. This is standard practice: you do not want to rebuild your Batch container image every time you edit a flow file. Metaflow injects the flow code at job submission time, not at image build time. But this means that at the moment the evaluate step’s container starts and Metaflow begins deserializing self.model_artifacts from S3, the Python interpreter in that container has no knowledge of flows.model_training_flow as a module. The pickle artifact encodes the expectation that this module exists and exports FeatureEncoder. The expectation is never met.

How pickle module-path serialization causes cross-container deserialization failures

Python’s pickle protocol serializes class instances by recording two pieces of information: the fully qualified module path of the class (e.g., flows.model_training_flow.FeatureEncoder) and the instance’s __dict__ (e.g., {"categories": [...], "encoder_map": {...}}). Pickle does not serialize the class definition itself — it does not include the source code of FeatureEncoder.__init__ or FeatureEncoder.encode. When you deserialize the pickle bytes in a different Python process, the unpickler executes the equivalent of import flows.model_training_flow; cls = flows.model_training_flow.FeatureEncoder and then reconstructs the instance by calling cls.__new__(cls) and updating its __dict__. If the import fails, the entire deserialization fails. There is no fallback. The error is not recoverable at runtime.

Metaflow uses pickle as its default artifact serialization protocol because pickle handles the full range of Python objects that a machine learning step might produce: trained model objects, scikit-learn transformers, numpy arrays, custom container objects. Metaflow stores the pickle bytes as an S3 object under the run’s artifact prefix and retrieves it when a downstream step accesses the corresponding self.* attribute. The deserialization happens transparently — the step author does not write any deserialization code; they simply access self.model_artifacts and Metaflow handles the S3 fetch and unpickling. This transparency is exactly why the failure mode is subtle: the error manifests as a single line in the evaluate step (artifacts = self.model_artifacts), not as explicit deserialization code, and the error message names a module path (flows.model_training_flow) rather than a class the engineer remembers writing.

# Diagnostic: how to confirm the pickle module path and reproduce the error

# 1. Query Metaflow run history to count failed evaluate steps
from metaflow import Flow, namespace

namespace(None)  # access all namespaces
flow = Flow("ModelTrainingFlow")

failed_runs = []
for run in flow.runs():
    evaluate_step = run["evaluate"]
    for task in evaluate_step.tasks():
        if not task.successful:
            failed_runs.append({
                "run_id": run.id,
                "pathspec": task.pathspec,
                "finished_at": task.finished_at,
            })

print(f"Failed evaluate tasks: {len(failed_runs)}")
# Output: Failed evaluate tasks: 12
# All 12 weekly runs since 2026-07-14 have a failed evaluate step

# 2. Inspect the pickle bytes to confirm the serialized module path
import pickle, io

# Fetch the artifact bytes from the most recent successful training step
training_run = next(r for r in flow.runs() if r["training"].task.successful)
artifact_bytes = training_run["training"].task.data._artifacts["model_artifacts"]._load()

# Scan the pickle bytes for the module path reference (human-readable in pickle protocol 2+)
# pickle stores class references as: b'\x8c\x1cflows.model_training_flow\x8c\x0eFeatureEncoder'
# Quick confirmation: grep for 'flows' in the pickle bytes
print("flows" in str(artifact_bytes))
# Output: True

# 3. Reproduce the ModuleNotFoundError outside of Metaflow
# (Run this in a Python environment that does NOT have flows/ in sys.path)
import sys
print([p for p in sys.path if "flows" in p])
# Output: []  — flows/ is not in the Python path

try:
    obj = pickle.loads(artifact_bytes)
except ModuleNotFoundError as e:
    print(f"Confirmed: {e}")
# Output: Confirmed: No module named 'flows'

# 4. Check CloudWatch Logs for the evaluate step's Batch job
# aws logs filter-log-events \
#   --log-group-name /aws/batch/job \
#   --filter-pattern "ModuleNotFoundError" \
#   --start-time $(date -d "2026-07-14" +%s000)
# Returns 12 log events — one per weekly run — each with:
# Traceback (most recent call last):
#   File "/opt/conda/lib/python3.11/site-packages/metaflow/datastore/datastore.py", line ...
#   ...
# ModuleNotFoundError: No module named 'flows'

The manual workaround the senior ML engineer had developed over 12 weeks was technically correct but operationally unsustainable. After each training run, the engineer would: (1) identify the run ID from the Metaflow UI; (2) download the model_artifacts pickle from S3 using aws s3 cp; (3) run a local Python script that added flows/ to sys.path, imported FeatureEncoder from the flow file, deserialized the artifact, converted the FeatureEncoder instance to a plain serializable dict ({"categories": encoder.categories, "encoder_map": encoder.encoder_map}), and re-serialized the artifact as a new pickle; (4) re-upload the modified artifact to S3 at the same key; (5) manually re-trigger the evaluate step using python flows/model_training_flow.py resume <run_id>. The process took approximately 45 minutes per training run. Over 12 weekly runs, that was 9 hours of engineering time spent on a workaround for a structural bug — time that was logged inconsistently and never surfaced as a bug ticket.

October 13: new ML engineer asks why evaluate always fails on first run

The question from the new ML engineer on October 13 had an unusual diagnostic quality: they asked “why does it always fail on first run” rather than “why did it fail this run.” Because they were new, they had no prior context that would cause them to accept the failure as normal. The question was routed to the Metaflow developer retainer the same afternoon. The retainer investigation began with the Metaflow Client API to establish the scope of the failure — specifically, how many runs had failed, when the failures had started, and whether any run had succeeded without manual intervention. The Metaflow Client API query confirmed 12 consecutive failed evaluate steps, all starting on July 14, 2026. A review of the git history for flows/model_training_flow.py showed that the FeatureEncoder class and the updated self.model_artifacts assignment were introduced in a commit on July 14, 2026 with the message “add categorical encoding to training step.” The correlation was immediate and unambiguous.

The fix required three coordinated changes: extracting FeatureEncoder to a proper Python package that would be installed in the Docker image, updating requirements.txt to include the new package, and rebuilding the Docker image. A fourth change updated the flow to import FeatureEncoder from the installed package rather than defining it inline. The package extraction took 2 hours: creating the package structure, writing tests for FeatureEncoder, adding a setup.py, and validating that the package was importable in a clean Python environment. The Docker image rebuild took 30 minutes and was coordinated with the infrastructure team. The flow rewrite and test run took 1.5 hours.

# Fix step 1: extract FeatureEncoder to a proper installable package
# Package structure:
#
# summit_ridge_ml_utils/
#   __init__.py
#   encoding.py
#   setup.py
#   requirements.txt  (this is the package's own requirements, not the Docker image's)

# summit_ridge_ml_utils/encoding.py
class FeatureEncoder:
    """Categorical feature encoder for Summit Ridge ML models.

    Encodes string category values to integer indices. Unknown values
    return -1. Serialization-safe: no dependency on any flow file module path.
    Instances of this class can be pickled and unpickled in any Python
    environment that has summit_ridge_ml_utils installed.
    """
    def __init__(self, categories):
        self.categories = list(categories)
        self.encoder_map = {cat: i for i, cat in enumerate(self.categories)}

    def encode(self, value):
        return self.encoder_map.get(value, -1)

    def encode_series(self, series):
        return series.map(lambda v: self.encoder_map.get(v, -1))

    def __repr__(self):
        return f"FeatureEncoder(n_categories={len(self.categories)})"


# summit_ridge_ml_utils/__init__.py
from .encoding import FeatureEncoder

__all__ = ["FeatureEncoder"]
__version__ = "0.1.0"


# summit_ridge_ml_utils/setup.py
from setuptools import setup, find_packages

setup(
    name="summit_ridge_ml_utils",
    version="0.1.0",
    packages=find_packages(),
    python_requires=">=3.9",
    install_requires=[],  # no external dependencies — pure Python
)

# Fix step 2: add summit_ridge_ml_utils to the Docker image's requirements.txt
# requirements.txt (Docker image — summit-ridge-ml:latest)
# ... existing dependencies ...
metaflow>=2.11.0
scikit-learn>=1.3.0
pandas>=2.0.0
numpy>=1.24.0
# NEW: add summit_ridge_ml_utils so FeatureEncoder is importable in all Batch containers
summit_ridge_ml_utils @ git+https://github.com/summit-ridge/summit_ridge_ml_utils.git@v0.1.0

# Fix step 3: rebuild and push Docker image to ECR
# docker build -t summit-ridge-ml:latest .
# docker tag summit-ridge-ml:latest 123456789012.dkr.ecr.us-east-1.amazonaws.com/summit-ridge-ml:latest
# docker push 123456789012.dkr.ecr.us-east-1.amazonaws.com/summit-ridge-ml:latest

# Fix step 4: update flows/model_training_flow.py to import FeatureEncoder from the package
# flows/model_training_flow.py — fixed version

from metaflow import FlowSpec, step, batch, retry
import numpy as np
import pandas as pd

# FIX: import FeatureEncoder from the installed package, not defined inline
# summit_ridge_ml_utils is installed in the Docker image via requirements.txt
# pickle will now serialize FeatureEncoder as: summit_ridge_ml_utils.encoding.FeatureEncoder
# This module IS importable in all Batch containers because it is in requirements.txt
from summit_ridge_ml_utils import FeatureEncoder


class ModelTrainingFlow(FlowSpec):

    @step
    def start(self):
        self.next(self.training)

    @batch(cpu=4, memory=32000, image="summit-ridge-ml:latest")
    @retry(times=2)
    @step
    def training(self):
        from sklearn.linear_model import LogisticRegression
        from sklearn.preprocessing import StandardScaler

        df = pd.read_parquet("s3://summit-ridge-data/features/latest.parquet")
        categories = df["product_category"].unique().tolist()

        # FIX: FeatureEncoder is now from summit_ridge_ml_utils (an installed package)
        # pickle will serialize this as summit_ridge_ml_utils.encoding.FeatureEncoder
        # Any Batch container with summit_ridge_ml_utils in requirements.txt can deserialize it
        encoder = FeatureEncoder(categories)
        X = encoder.encode_series(df["product_category"]).values.reshape(-1, 1)
        y = df["converted"].values

        scaler = StandardScaler()
        X_scaled = scaler.fit_transform(X)

        model = LogisticRegression()
        model.fit(X_scaled, y)

        self.model_artifacts = {
            "model": model,
            "encoder": encoder,   # now serializes as summit_ridge_ml_utils.encoding.FeatureEncoder
            "scaler": scaler,
            "categories": categories,
        }
        self.next(self.evaluate)

    @batch(cpu=2, memory=8000, image="summit-ridge-ml:latest")
    @step
    def evaluate(self):
        # FIX: self.model_artifacts now deserializes correctly because
        # summit_ridge_ml_utils.encoding.FeatureEncoder is importable
        # in this container — no ModuleNotFoundError

        artifacts = self.model_artifacts   # deserializes without error
        encoder = artifacts["encoder"]     # FeatureEncoder instance fully functional
        model = artifacts["model"]
        scaler = artifacts["scaler"]

        df_test = pd.read_parquet("s3://summit-ridge-data/features/test.parquet")
        X_test = encoder.encode_series(df_test["product_category"]).values.reshape(-1, 1)
        X_test_scaled = scaler.transform(X_test)

        self.accuracy = model.score(X_test_scaled, df_test["converted"].values)
        self.next(self.end)

    @step
    def end(self):
        print(f"Evaluation accuracy: {self.accuracy:.4f}")


if __name__ == "__main__":
    ModelTrainingFlow()

The test run on October 13 confirmed that the evaluate step completed without error for the first time since July 14, 2026. The model_artifacts artifact deserialized correctly: FeatureEncoder was now serialized as summit_ridge_ml_utils.encoding.FeatureEncoder, and the evaluate step’s Batch container could import it from the installed package. The self.accuracy attribute was populated with the correct evaluation metric and the flow reached the end step. The 5.5-hour engagement covered 1 hour of investigation (Metaflow Client API run history query confirming 12 failed evaluate steps since July 14; git history review confirming FeatureEncoder introduction on July 14; pickle byte inspection confirming flows.model_training_flow.FeatureEncoder module path; Docker image manifest review confirming flows/ directory absent), 2 hours of package extraction (creating summit_ridge_ml_utils package structure; writing FeatureEncoder with proper docstrings; adding encode_series convenience method; writing unit tests; creating setup.py; pushing to the internal GitHub repository with tag v0.1.0; coordinating with infrastructure for the Docker image rebuild), 1.5 hours of flow rewrite and test run (flows/model_training_flow.py updated to import from summit_ridge_ml_utils; test run confirmed evaluate step success; self.accuracy metric confirmed correct), and 1 hour of team documentation (documenting the root cause in Confluence; adding a “Metaflow Artifact Serialization Policy” section to the ML platform runbook: all classes stored as self.* artifacts must be defined in packages listed in requirements.txt; no inline class definitions in flow files for any object stored as an artifact).

Why inline class definitions in Metaflow artifacts are a recurring retainer pattern

The Summit Ridge ML Systems incident demonstrates a failure class that is structural to how Metaflow’s artifact serialization interacts with AWS Batch container isolation. Metaflow’s design separates the flow execution environment (the Docker image, installed in the Batch container) from the flow definition (the Python file, injected at job submission). This separation is intentional and correct: it means you can modify a flow file without rebuilding a Docker image, which is the vast majority of day-to-day flow development work. But this separation creates a gap for class definitions: if a class is defined in the flow file (part of the injected flow definition) rather than in an installed package (part of the Docker image), that class is present in the training step’s Python environment but not in the evaluate step’s Python environment. When pickle serializes an instance of that class, it records the module path from the training environment. When Metaflow deserializes the artifact in the evaluate environment, Python tries to import the module path and fails because the flow file is not a module in the installed sense.

The failure is not detected at write time — self.model_artifacts = {...} succeeds, the pickle bytes are written to S3, and the training step completes successfully. The failure surfaces only in the first downstream step that reads the artifact. In a pipeline where the training step is computationally expensive (4 CPU, 32 GB memory, potentially hours of training time) and the evaluate step is the very next step, this means the expensive compute work completes successfully and only then does the pipeline fail. The retry behavior (@retry(times=2) on the training step) does not help: the training step is not failing, the evaluate step is. The failure will repeat on every retry of the evaluate step because the serialized artifact is already in S3 and will not change between retries.

The normalized manual workaround is the most dangerous aspect of this failure pattern. Once an engineer develops a reliable workaround and the pipeline continues producing results (with a 45-minute delay each week), there is no operational pressure to investigate the root cause. New team members join and are told the workaround is “a known issue.” The workaround is not logged as a bug or a retainer hour because it is simply “how we run the pipeline.” A quarterly Metaflow artifact serialization audit — reviewing all self.* assignments in all flow files for inline class definitions or local module references, and verifying that all classes stored as artifacts are importable from packages in requirements.txt — catches this class of failure before it normalizes into a manual process.

How HourTab helps Metaflow developer retainers

The Summit Ridge ML Systems engagement totaled 5.5 hours: investigation (1h — 0.25h Metaflow Client API run history query confirming 12 failed evaluate steps since July 14; 0.25h git log review confirming FeatureEncoder introduction on July 14 2026 as the causal commit; 0.25h pickle byte inspection using artifact bytes from last successful training task, confirming flows.model_training_flow.FeatureEncoder module path in pickle bytes; 0.25h Docker image manifest review confirming absence of flows/ directory), package extraction and Docker image update (2h — 0.5h creating summit_ridge_ml_utils package with encoding.py, __init__.py, setup.py, and unit tests; 0.25h internal GitHub repository setup and v0.1.0 tag; 0.25h requirements.txt update with summit_ridge_ml_utils @ git+https://... reference; 0.5h coordinating Docker image rebuild and ECR push with infrastructure team; 0.5h verifying that summit_ridge_ml_utils.encoding.FeatureEncoder is importable in a fresh container from the rebuilt image), flow rewrite and test run (1.5h — 0.25h flows/model_training_flow.py updated to import FeatureEncoder from summit_ridge_ml_utils; 0.25h encode_series convenience method added and training step updated to use it; 0.5h full test run of ModelTrainingFlow confirming evaluate step success; 0.25h confirming self.accuracy metric value against expected range; 0.25h confirming that the new pickle bytes serialize as summit_ridge_ml_utils.encoding.FeatureEncoder), team documentation (1h — 0.5h Confluence runbook: Metaflow Artifact Serialization Policy documenting the inline class definition prohibition and the required package extraction process; 0.5h Slack #ml-platform post explaining the root cause and fix for current team awareness).

In a HourTab retainer log, the incident session reads: “ModelTrainingFlow evaluate step ModuleNotFoundError investigation: Metaflow 2.11.5 on AWS Batch; training step @batch(cpu=4, memory=32000) stores FeatureEncoder in self.model_artifacts; FeatureEncoder defined inline in flows/model_training_flow.py; pickle serializes as flows.model_training_flow.FeatureEncoder; evaluate step Batch container built from requirements.txt only, no flows/ directory; ModuleNotFoundError: No module named ‘flows’ on every run since July 14 2026; 12 weekly runs affected; 45 min manual workaround per run (9h total); fix: summit_ridge_ml_utils package extraction (encoding.py + setup.py + GitHub v0.1.0 tag); requirements.txt updated; Docker image rebuilt and pushed to ECR; flows/model_training_flow.py imports FeatureEncoder from summit_ridge_ml_utils; test run confirms evaluate step success; 5.5h.”

HourTab turns a time-tracker CSV into a public retainer-hours URL your client can bookmark — no login, no portal, no “how many hours do I have left?” emails. Free for one retainer. $9/mo for up to ten.

FAQ: Metaflow developer retainers

See also: Apache Airflow developer retainer · Databricks developer retainer · Apache Spark developer retainer