Kubeflow developer on retainer: Kubeflow 1.7→1.8 upgrade changes model artifact base URI so pipeline evaluation step silently returns NaN metrics for 6 weeks — Kubeflow UI marks step Succeeded — discovered when ML engineer investigates model monitoring dashboard — Kubeflow developer on monthly retainer

Kubeflow 1.7.3→1.8.1 • KFP SDK 2.0.1 • Python 3.11 • GKE 1.27 • GCS artifact storage • @component decorator • Output[Model] • Input[Model] • pipeline_root URI mismatch • FileNotFoundError caught silently • NaN evaluation metrics • 6 weeks undetected • retainer billing

On October 16, 2026, an ML engineer at Coastal ML Systems Corp. opened the model monitoring dashboard for the weekly production model evaluation pipeline and noticed something that had been hidden in plain sight since August 4: every evaluation metric for the past six weeks — accuracy, AUC, F1, and RMSE — was NaN. The Kubeflow UI showed all pipeline steps as Succeeded for each of the six weekly runs. The evaluation component had been returning NaN metrics quietly, its underlying FileNotFoundError swallowed by a broad except block, while Kubeflow’s execution engine recorded the exit code 0 and marked the step as Succeeded. The Kubeflow retainer developer was engaged on October 17. The root cause was a Kubeflow cluster upgrade from 1.7.3 to 1.8.1 in July 2026 that had changed the model artifact base URI from gs://coastal-ml-pipelines/pipeline_root/ to gs://coastal-ml-pipelines/pipeline_root/v2/ — a path that previously compiled and scheduled pipeline runs did not know about.

Coastal ML Systems Corp. ran a weekly model evaluation pipeline for a fraud detection system. The pipeline consisted of three Kubeflow components: train_model (trains a gradient boosting classifier on the past 90 days of transaction data), evaluate_model (computes accuracy, AUC, F1, and RMSE on a held-out validation set), and register_model (conditionally promotes the model to the serving registry if AUC exceeds 0.93). The pipeline ran on GKE 1.27 with Kubeflow Pipelines 1.7.3 and used a GCS bucket (gs://coastal-ml-pipelines/) as the artifact storage backend. The pipeline_root was configured as gs://coastal-ml-pipelines/pipeline_root/. The pipeline had been running correctly every week from April through July 2026, producing evaluation metrics consistent with a well-maintained fraud detection model.

On July 14, 2026, the infrastructure team upgraded the GKE cluster to a newer GKE version, and as part of the upgrade, Kubeflow Pipelines was upgraded from 1.7.3 to 1.8.1. The upgrade was completed successfully — the Kubeflow UI was accessible, new pipeline runs could be submitted, and the upgrade changelog was reviewed by the infrastructure team lead. What the changelog did not prominently surface was that Kubeflow 1.8 changed the default subdirectory structure for pipeline artifact storage, adding a v2/ subdirectory under the configured pipeline_root.

# pipeline.py — Coastal ML Systems Corp.
# KFP SDK 2.0.1, Python 3.11
# Compiled and scheduled May 2026; not recompiled after Kubeflow upgrade.

from kfp import dsl
from kfp.dsl import Input, Output, Model, Dataset, Artifact
from typing import NamedTuple

@dsl.component(
    base_image="gcr.io/coastal-ml/fraud-trainer:v1.4",
    packages_to_install=["lightgbm==4.1.0", "scikit-learn==1.3.2"]
)
def train_model(
    training_data: Input[Dataset],
    model: Output[Model],
    n_estimators: int = 500
):
    import lightgbm as lgb
    # ... training logic ...
    booster.save_model(model.path + "/model.txt")
    # After Kubeflow 1.8 upgrade, model.path resolves to:
    # gs://coastal-ml-pipelines/pipeline_root/v2///train_model/model/
    # (new /v2/ subdirectory added by Kubeflow 1.8)

@dsl.component(
    base_image="gcr.io/coastal-ml/fraud-evaluator:v1.2",
    packages_to_install=["lightgbm==4.1.0", "scikit-learn==1.3.2"]
)
def evaluate_model(
    validation_data: Input[Dataset],
    model: Input[Model],
    metrics: Output[Artifact]
) -> NamedTuple("Outputs", [("accuracy", float), ("auc", float),
                             ("f1", float), ("rmse", float)]):
    import lightgbm as lgb
    from collections import namedtuple

    # BUG: broad except block catches FileNotFoundError silently.
    # When model.path resolves to a GCS path that doesn't exist
    # (old /pipeline_root/ scheme instead of new /pipeline_root/v2/),
    # lgb.Booster(model_file=...) raises FileNotFoundError.
    # The except block catches it, prints a warning, and returns NaN.
    # Kubeflow execution engine sees exit code 0 → marks step Succeeded.
    try:
        booster = lgb.Booster(model_file=model.path + "/model.txt")
        # ... evaluation logic ...
        accuracy = float(accuracy_score(y_val, y_pred))
        auc = float(roc_auc_score(y_val, y_prob))
        f1 = float(f1_score(y_val, y_pred))
        rmse = float(mean_squared_error(y_val, y_prob, squared=False))
    except Exception as e:
        # BUG: this except block catches FileNotFoundError and returns
        # NaN metrics without re-raising. Kubeflow sees exit code 0
        # and marks the step as Succeeded. The NaN metrics are logged
        # to the Kubeflow UI metrics panel as valid metric values.
        print(f"Warning: model evaluation failed: {e}")  # goes to pod logs
        accuracy, auc, f1, rmse = float("nan"), float("nan"), float("nan"), float("nan")

    Outputs = namedtuple("Outputs", ["accuracy", "auc", "f1", "rmse"])
    return Outputs(accuracy=accuracy, auc=auc, f1=f1, rmse=rmse)

@dsl.pipeline(name="fraud-detection-weekly-eval")
def fraud_evaluation_pipeline(pipeline_root: str):
    train_op = train_model(training_data=..., n_estimators=500)
    eval_op = evaluate_model(
        validation_data=...,
        model=train_op.outputs["model"]
    )
    # register_model conditionally based on AUC — but NaN > 0.93 is False,
    # so no new models are promoted to serving either.

# KFP client configuration at compile time (May 2026, before upgrade):
# pipeline_root = "gs://coastal-ml-pipelines/pipeline_root/"
# After Kubeflow 1.8 upgrade, new runs write to:
# "gs://coastal-ml-pipelines/pipeline_root/v2/"
# The compiled pipeline specification still references the old base URI.

After the July 14 Kubeflow upgrade, the train_model component began writing its model artifact to the new /v2/ URI scheme: gs://coastal-ml-pipelines/pipeline_root/v2/fraud-detection-weekly-eval/<run_id>/train_model/model/. The evaluate_model component, however, received the model artifact path via Input[Model] from the compiled pipeline specification, which had been compiled in May 2026 before the upgrade. The compiled specification referenced the old pipeline_root without the /v2/ subdirectory. When the evaluation component attempted to load the model from gs://coastal-ml-pipelines/pipeline_root/fraud-detection-weekly-eval/<run_id>/train_model/model/model.txt, the GCS path did not exist — the model was actually at the /v2/ path. The lgb.Booster(model_file=...)’s internal GCS client raised FileNotFoundError.

The evaluate_model component’s broad except Exception as e block caught the error, printed a warning to stdout (which was captured in the component’s Kubernetes pod logs), and returned NaN for all four evaluation metrics. The component function then exited normally with exit code 0. Kubeflow’s pipeline execution engine, which determines step success or failure by the container exit code, recorded the step as Succeeded. The NaN metrics were logged to the Kubeflow UI’s artifact metadata panel as valid metric values — Kubeflow has no built-in validation that metric values are non-NaN.

# Diagnostic: trace artifact URI mismatch after Kubeflow 1.8 upgrade
# Run October 17, 2026 as part of Kubeflow retainer investigation

# Step 1: Check GCS bucket structure post-upgrade
# gsutil ls gs://coastal-ml-pipelines/pipeline_root/
# Output:
#   gs://coastal-ml-pipelines/pipeline_root/v2/           ← new, created July 14
#   gs://coastal-ml-pipelines/pipeline_root/fraud-.../    ← old runs (before upgrade)

# Step 2: Inspect August 4 pipeline run (first run after upgrade)
run_id = "fraud-detection-weekly-eval-2026-08-04-07b3c2"

# Check where training component wrote the model artifact:
# gsutil ls gs://coastal-ml-pipelines/pipeline_root/v2/fraud-detection-weekly-eval/{run_id}/train_model/model/
# Output: model.txt (correct — training component wrote here)

# Check where evaluation component tried to read the model:
# gsutil ls gs://coastal-ml-pipelines/pipeline_root/fraud-detection-weekly-eval/{run_id}/train_model/model/
# Output: CommandException: No URLs matched: gs://coastal-ml-pipelines/pipeline_root/fraud...
# (This path does not exist — model is at /v2/ subdirectory, not here)

# Step 3: Inspect evaluation component pod logs for August 4 run
# kubectl logs pod/evaluate-model- -n kubeflow
# Output:
#   2026-08-04T07:31:22Z INFO Starting model evaluation
#   2026-08-04T07:31:24Z WARNING Warning: model evaluation failed:
#     [Errno 2] No such file or directory:
#     '/tmp/inputs/model/data/model.txt'
#   (LightGBM mounts GCS artifact to /tmp/inputs/ directory)
#   2026-08-04T07:31:24Z INFO Evaluation complete: accuracy=nan, auc=nan, f1=nan, rmse=nan
# Exit code: 0 (because except block ran without re-raising)

# Step 4: Compare compiled pipeline_root with cluster configuration
# KFP client pipeline_root from compiled spec (May 2026):
#   gs://coastal-ml-pipelines/pipeline_root/
# KFP server pipeline_root from cluster config (after July 14 upgrade):
#   gs://coastal-ml-pipelines/pipeline_root/v2/
# Gap: compiled spec does not include /v2/ subdirectory

# Step 5: Verify 6 affected runs (August 4 through September 15)
# All 6 runs show: train_model=Succeeded, evaluate_model=Succeeded
# All 6 runs show metrics: accuracy=NaN, auc=NaN, f1=NaN, rmse=NaN
# All 6 runs: register_model step NOT executed (NaN > 0.93 == False)

Why Succeeded does not mean correct in Kubeflow Pipelines

Kubeflow Pipelines determines component success or failure by the container exit code: exit code 0 means Succeeded, any non-zero exit code means Failed. This design is consistent with standard Unix process semantics and works correctly when component implementations propagate exceptions to the process exit code. The failure point is when a component catches an exception without re-raising it: the except Exception as e block runs to completion, the function returns normally (including returning NaN values), and the container exits with code 0. From Kubeflow’s perspective, the component ran successfully. The NaN metric values are treated as valid float outputs and stored in the artifact metadata database.

The Kubeflow UI’s pipeline run view displays step status (Succeeded, Failed, Running, Skipped) and execution time. It does not validate metric values, compare them against historical ranges, or flag NaN as a sentinel error condition. An engineer monitoring the Kubeflow UI for step status — which is the most common monitoring pattern — would see all steps as Succeeded and have no indication that the evaluation step had silently failed to load the model artifact.

The six-week detection gap arose from a combination of three factors: the broad except block that caught the error without re-raising; the Kubeflow UI showing Succeeded based on exit code rather than metric validity; and the monitoring practice of checking step status rather than metric values. The model monitoring dashboard that finally surfaced the issue (because it tracked the metric values over time, not just step status) was the only system that would have caught the problem — and it was checked infrequently.

The fix: update pipeline_root and re-raise exceptions

The fix had two components. First, the pipeline_root URI in the KFP client configuration was updated to include the /v2/ subdirectory, and all three pipeline specifications were recompiled with the corrected pipeline_root. The old scheduled runs were deleted and new scheduled runs were registered using the recompiled pipeline specifications. Second, the evaluate_model component’s error handling was updated to log the exception and then re-raise it, so that future FileNotFoundError events would cause the component to exit with a non-zero exit code and be marked as Failed in the Kubeflow UI.

# pipeline.py — fixed version, deployed October 17, 2026
# KFP SDK 2.0.1, Python 3.11, Kubeflow Pipelines 1.8.1

import logging
logger = logging.getLogger(__name__)

@dsl.component(
    base_image="gcr.io/coastal-ml/fraud-evaluator:v1.3",  # bumped version
    packages_to_install=["lightgbm==4.1.0", "scikit-learn==1.3.2"]
)
def evaluate_model(
    validation_data: Input[Dataset],
    model: Input[Model],
    metrics: Output[Artifact]
) -> NamedTuple("Outputs", [("accuracy", float), ("auc", float),
                             ("f1", float), ("rmse", float)]):
    import lightgbm as lgb
    import logging
    from collections import namedtuple

    logger = logging.getLogger(__name__)

    # FIX: log the error with severity ERROR, then re-raise.
    # Re-raising causes the component container to exit with code 1
    # (Python exception exit code), which Kubeflow marks as Failed.
    # The Failed status is visible in the Kubeflow UI immediately,
    # alerting the team that the evaluation step has a problem.
    try:
        booster = lgb.Booster(model_file=model.path + "/model.txt")
        # ... evaluation logic ...
        accuracy = float(accuracy_score(y_val, y_pred))
        auc = float(roc_auc_score(y_val, y_prob))
        f1 = float(f1_score(y_val, y_pred))
        rmse = float(mean_squared_error(y_val, y_prob, squared=False))
    except FileNotFoundError as e:
        # FIX: log with ERROR severity (captured in Kubeflow pod logs),
        # then re-raise so Kubeflow marks this step as Failed.
        logger.error(
            f"Model artifact not found at {model.path}: {e}. "
            f"Check pipeline_root URI matches cluster configuration "
            f"(Kubeflow upgrade may have added /v2/ subdirectory)."
        )
        raise  # FIX: re-raise instead of returning NaN

    Outputs = namedtuple("Outputs", ["accuracy", "auc", "f1", "rmse"])
    return Outputs(accuracy=accuracy, auc=auc, f1=f1, rmse=rmse)

# KFP client configuration — recompile all pipelines with updated pipeline_root:
client = kfp.Client(host="https://kubeflow.coastal-ml.internal")
pipeline_root = "gs://coastal-ml-pipelines/pipeline_root/v2/"  # FIX: added /v2/

# Recompile pipeline with updated pipeline_root
kfp.compiler.Compiler().compile(
    pipeline_func=fraud_evaluation_pipeline,
    package_path="fraud_evaluation_pipeline_v2.yaml",
    pipeline_parameters={"pipeline_root": pipeline_root}
)

# Delete old scheduled runs (using old compiled spec with incorrect pipeline_root)
# client.delete_recurring_run(recurring_run_id="old-recurring-run-id")

# Register new scheduled runs with updated pipeline spec
client.create_recurring_run(
    experiment_id=experiment_id,
    job_name="fraud-detection-weekly-eval-fixed",
    pipeline_package_path="fraud_evaluation_pipeline_v2.yaml",
    cron_expression="0 7 * * 1",  # Every Monday at 07:00 UTC
    pipeline_root=pipeline_root
)

# Validation run (October 17, 2026, manual trigger):
# train_model: Succeeded (wrote model to /v2/ path)
# evaluate_model: Succeeded (read model from /v2/ path correctly)
# Metrics: accuracy=0.921, auc=0.947, f1=0.883, rmse=0.142
# (consistent with pre-upgrade values from July 7, 2026:
#  accuracy=0.918, auc=0.943, f1=0.879, rmse=0.148)

The validation run completed successfully on October 17. The evaluate_model component loaded the model artifact from the correct /v2/ URI, computed the evaluation metrics, and returned accuracy=0.921, AUC=0.947, F1=0.883, and RMSE=0.142 — values consistent with the pre-upgrade metrics from July 7. The register_model step executed (AUC=0.947 > 0.93 threshold) and promoted the newly trained model to the serving registry. The model monitoring dashboard was also updated to send an alert if any evaluation metric was NaN on the next pipeline run, as an additional layer of detection.

Why Kubeflow artifact URI mismatch after upgrade is a recurring retainer pattern

The Kubeflow 1.7→1.8 pipeline_root URI change is an instance of a failure class that occurs at the intersection of silent exception handling and monitoring gap. In well-engineered Kubeflow pipelines, components propagate exceptions to the process exit code by re-raising after logging — ensuring that Kubeflow’s execution engine sees the failure and marks the step as Failed. In pipelines developed under time pressure or by engineers unfamiliar with Kubeflow’s exit-code-based status reporting, broad except blocks that return default values (NaN, empty lists, zero) are common. These patterns are invisible during normal operation but become failure traps when the environment changes — a Kubeflow upgrade, a GCS bucket migration, or a pipeline_root reconfiguration.

The monitoring gap is equally important. The Kubeflow UI’s step status display (Succeeded/Failed) is the most visible monitoring surface and the one most commonly checked. Metric value monitoring — verifying that evaluation metrics are non-NaN and within expected historical ranges — requires either a separate monitoring dashboard or a deliberate practice of checking metric panels (not just step status) in the Kubeflow UI. A Kubeflow retainer developer who implements a post-upgrade validation checklist (submit a validation run immediately after any Kubeflow upgrade; verify non-NaN evaluation metrics in the Kubeflow UI metrics panel; confirm artifact URIs by listing the GCS bucket before and after the upgrade) catches this class of failure within 24 hours of the upgrade rather than six weeks later.

The 5.0-hour engagement covered investigation (2.0h — 0.5h Kubeflow UI review: all 6 weekly runs since August 4 show Succeeded but all evaluation metrics are NaN; 0.5h GCS bucket audit: gsutil ls gs://coastal-ml-pipelines/pipeline_root/ shows /v2/ subdirectory created July 14 (upgrade date); 0.5h evaluation component pod log review: Warning: model evaluation failed: [Errno 2] No such file or directory — silent catch of FileNotFoundError; 0.5h compiled pipeline specification review: pipeline_root set to old URI without /v2/), fix implementation (2.0h — 0.5h pipeline_root URI updated to include /v2/ in KFP client config; 0.5h evaluate_model component updated: except FileNotFoundError as e: logger.error(...); raise; 0.5h all 3 pipeline specifications recompiled; 0.5h old scheduled runs deleted, new scheduled runs registered), and validation (1.0h — 0.5h manual validation pipeline run: accuracy=0.921, AUC=0.947, F1=0.883, RMSE=0.142; 0.25h comparison with pre-upgrade metrics (July 7): within normal variation; 0.25h model monitoring dashboard alert configured for NaN metric values).

In a HourTab retainer log, the incident session reads: “Kubeflow pipeline evaluation NaN metrics investigation: Kubeflow 1.7.3→1.8.1, KFP SDK 2.0.1, Python 3.11, GKE 1.27, GCS; fraud-detection-weekly-eval pipeline; July 14 2026 Kubeflow upgrade changes pipeline_root artifact storage from gs://coastal-ml-pipelines/pipeline_root/ to gs://coastal-ml-pipelines/pipeline_root/v2/; training component writes model to /v2/ URI; evaluation component reads from old URI (pipeline compiled before upgrade); FileNotFoundError caught by broad except block: returns NaN for all 4 metrics; Kubeflow exit code 0 → UI shows Succeeded; 6 weekly pipeline runs (August 4 to September 15 2026) all NaN; discovered October 16 2026 when ML engineer checks monitoring dashboard; fix: pipeline_root URI updated to /v2/; pipelines recompiled and re-scheduled; evaluate_model re-raises FileNotFoundError after logging; validation run October 17: accuracy=0.921, AUC=0.947; 5.0h.”

HourTab turns a time-tracker CSV into a public retainer-hours URL your client can bookmark — no login, no portal, no “how many hours do I have left?” emails. Free for one retainer. $9/mo for up to ten.

FAQ: Kubeflow developer retainers

See also: Tecton developer retainer · W&B developer retainer · MLflow developer retainer