Argo Workflows developer on retainer: DAG template feature-engineering step missing dependencies: [preprocess] — reads /mnt/pipeline/features.parquet while preprocess still writing — ArrowInvalid: Parquet magic bytes not found in footer on three of five retry attempts — pipeline fails on high-volume days — Argo Workflows developer on monthly retainer

Argo Workflows 3.5.4 • Python 3.11 • EKS 1.28 • DAGTemplate • preprocess writes 18–26 GB Parquet (30–45 min) • feature-engineering starts concurrently • pyarrow.parquet.read_table() on incomplete file • ArrowInvalid: Parquet magic bytes not found • retryStrategy limit: 5, backoff: 5m • pipeline fails 6 of 21 days • 4.0h retainer engagement

On October 21, 2026, after three weeks of intermittent daily pipeline failures, a data engineering team at Cascade Data Systems Inc. engaged their Argo Workflows retainer developer. The pipeline was failing with ArrowInvalid: Parquet magic bytes not found in footer in the feature-engineering step — but only on some days, and with no apparent pattern. Six of the previous 21 days had required manual re-runs. The Argo Workflows pod logs showed the error consistently originating from a pyarrow.parquet.read_table() call in the feature-engineering step, but the Parquet file at /mnt/pipeline/features.parquet was intact on the EFS volume when the data engineers checked it after each failure. The retainer developer found the root cause in one hour: the feature-engineering step was missing dependencies: [preprocess] in the DAG template. It was starting concurrently with preprocess and reading a partially written Parquet file.

Cascade Data Systems ran a daily ML feature pipeline on Argo Workflows 3.5.4 deployed on EKS 1.28. The pipeline used a DAGTemplate with five steps: data-ingestion (pulls raw click events from S3 to EFS, ~2 minutes), then in parallel preprocess and feature-engineering, then validation and model-inference. In the developer’s intent, preprocess was supposed to run before feature-engineering: preprocess read the raw click events, cleaned them, aggregated them by user, and wrote the result to /mnt/pipeline/features.parquet, a file that grew to 18–26 GB depending on daily event volume. feature-engineering then read features.parquet and computed rolling window and interaction features for model inference. The developer had tested the pipeline with small synthetic data where preprocess completed in under 30 seconds, so the timing issue never surfaced during development.

# daily-ml-feature-pipeline.yaml — Cascade Data Systems Inc.
# Argo Workflows 3.5.4 on EKS 1.28
# Production DAGTemplate — October 2026

apiVersion: argoproj.io/v1alpha1
kind: Workflow
metadata:
  generateName: daily-ml-pipeline-
spec:
  entrypoint: daily-pipeline-dag

  volumeClaimTemplates:
  - metadata:
      name: pipeline-pvc
    spec:
      accessModes: [ReadWriteMany]
      storageClassName: efs-sc
      resources:
        requests:
          storage: 100Gi

  templates:
  - name: daily-pipeline-dag
    dag:
      tasks:
      - name: data-ingestion
        template: data-ingestion-tpl

      - name: preprocess
        dependencies: [data-ingestion]   # Correct: waits for data-ingestion
        template: preprocess-tpl
        # preprocess reads raw events, writes /mnt/pipeline/features.parquet
        # Runtime: 30–45 minutes (depends on daily event volume: 18–26 GB output)

      - name: feature-engineering
        dependencies: [data-ingestion]   # BUG: should be [preprocess], not [data-ingestion]
        template: feature-engineering-tpl
        # feature-engineering reads /mnt/pipeline/features.parquet written by preprocess
        # But dependencies only lists data-ingestion, so it starts concurrently with preprocess
        # On most days: reads partially written Parquet → ArrowInvalid

      - name: validation
        dependencies: [preprocess, feature-engineering]
        template: validation-tpl

      - name: model-inference
        dependencies: [validation]
        template: model-inference-tpl

In Argo Workflows’ DAGTemplate, a task becomes eligible to start the moment all tasks listed in its dependencies field reach the Succeeded phase. There is no implicit ordering between tasks that share a common ancestor: if both preprocess and feature-engineering list only data-ingestion as a dependency, they both start as soon as data-ingestion succeeds. In production, data-ingestion completed in approximately two minutes. Both preprocess and feature-engineering pods were scheduled within 30 seconds of each other. preprocess began writing /mnt/pipeline/features.parquet using PyArrow’s ParquetWriter (which writes the file incrementally: row groups first, then the footer containing the row group metadata and magic bytes). feature-engineering called pyarrow.parquet.read_table(“/mnt/pipeline/features.parquet”) approximately two minutes after preprocess started writing. At that point, preprocess had written the Parquet file header and several row groups but had not yet written the footer. PyArrow raised ArrowInvalid: Parquet magic bytes not found in footer. Either the file is corrupted or this is not a parquet file.

# Investigation — Argo Workflows retainer developer, October 21 2026
# Cascade Data Systems Inc., Argo Workflows 3.5.4, EKS 1.28

# Step 1: Check step start times across failure and success days
argo get daily-ml-pipeline-abc123 -o json | \
    jq '.status.nodes | to_entries[] | {name: .value.displayName, startedAt: .value.startedAt, finishedAt: .value.finishedAt}'

# Failure day (October 7, high volume — preprocess took 43 min):
# data-ingestion:    startedAt=08:00:02, finishedAt=08:02:11
# preprocess:        startedAt=08:02:40  (started 29s after data-ingestion)
# feature-engineering: startedAt=08:02:43  (started 32s after data-ingestion)
# CONCURRENT START CONFIRMED: feature-engineering starts 3 seconds after preprocess

# Success day (October 9, low volume — preprocess took 28 min):
# data-ingestion:    startedAt=08:00:04, finishedAt=08:02:09
# preprocess:        startedAt=08:02:38
# feature-engineering: startedAt=08:02:41
# feature-engineering retries at 08:07:41 (5m backoff)
# feature-engineering retries at 08:12:41 (5m backoff)
# feature-engineering retries at 08:17:41 (5m backoff)
# feature-engineering retries at 08:22:41 (5m backoff)
# feature-engineering retries at 08:27:41 (5m backoff) — preprocess finished at 08:30:38 ← STILL RUNNING
# feature-engineering: 5/5 retries FAILED. Workflow Failed.
# (October 9 was actually a failure day, not a success day — mislabeled by on-call)

# Failure pattern: all 6 failures occurred when preprocess took >27 minutes
# (5 retries × 5m backoff = 27 minutes retry window; preprocess >27 min = all retries fail)
# Success on days when preprocess finished within 25 minutes (all retries succeeded eventually)

# Step 2: Confirm the fix
# Add dependencies: [preprocess] to feature-engineering task:
# feature-engineering starts only after preprocess Succeeded
# ArrowInvalid eliminated: features.parquet is always complete before feature-engineering reads it

# Step 3: Verify correct runtime after fix
# Pipeline runtime: preprocess (32min avg) + feature-engineering (16min avg) = 48min avg
# (Previously: data-ingestion (2min) + max(preprocess, feature-engineering_retries) = variable)

The fix was a single-line change in the DAGTemplate: updating feature-engineering’s dependencies from [data-ingestion] to [preprocess]. This ensured that feature-engineering would not start until preprocess had completed successfully and /mnt/pipeline/features.parquet was fully written with a valid Parquet footer. After the fix, the pipeline ran without a failure for 12 consecutive days. The average pipeline runtime normalized to 48 minutes (sequential preprocess + feature-engineering) from the previous erratic 50–90+ minute range on success days (retry overhead) and outright failures on high-volume days.

The retainer developer also implemented two prevention measures. First, an integration test was added to the CI pipeline that runs the DAGTemplate with synthetic data and verifies, via Argo Workflow event timestamps, that feature-engineering’s startedAt is always after preprocess’s finishedAt. Second, the team migrated the intermediate file handoff from a shared PVC path to Argo artifact outputs: preprocess now declares features.parquet as an artifact output, and feature-engineering declares it as an artifact input. Argo’s artifact dependency system enforces the producer-consumer ordering automatically, making the issue architecturally impossible regardless of what the dependencies field says.

Why missing DAG dependencies are a recurring Argo Workflows retainer pattern

Argo Workflows’ DAGTemplate requires explicit dependency declaration: the developer must list every step that must complete before the current step starts. This is correct and explicit — Argo cannot infer data-flow dependencies from shared PVC paths. The implication is that any implicit assumption about ordering (a step that writes data before another step reads it) must be expressed as an explicit dependencies entry. Forgetting to add a dependency is especially easy when two steps share a common ancestor: the developer creates both steps with dependencies: [data-ingestion] and mentally reads this as “both run after data ingestion” without noticing that the second step also requires the first step to complete.

The failure is intermittent because its manifestation depends on the relative timing of the concurrent steps, which depends on daily data volume. On days when the upstream step runs faster than the retry window allows, the pipeline succeeds with extra latency. On days when the upstream step runs slower than the retry window allows, the pipeline fails. This volume-correlated intermittency makes the failure look like a transient infrastructure issue — the ArrowInvalid: Parquet magic bytes not found error reads like data corruption, not a race condition. The data engineer who checks the EFS volume after a failure finds a valid Parquet file (because by the time they check, preprocess has finished writing). The evidence of the race is gone by the time the investigation begins.

An Argo Workflows retainer developer who compares step startedAt timestamps across failure and success days catches this pattern in one hour. The step start times tell the complete story: concurrent starts, retry exhaustion, volume-correlated failure rate. The fix takes five minutes. The prevention (artifact dependency migration) takes two hours and makes the topology error architecturally impossible on future pipelines.

The 4.0-hour engagement covered investigation (1.5h — 0.5h Argo Workflow status review: concurrent start times on all failure days; 0.5h correlation: step start deltas vs. preprocess runtime vs. failure/success; 0.5h root cause confirmation: ArrowInvalid from partial Parquet + concurrent start = missing dependency), fix implementation (1.0h — 0.5h dependencies: [preprocess] added to feature-engineering; 0.5h verified on next pipeline run: sequential start, no ArrowInvalid), and prevention (1.5h — 0.5h integration test: verify feature-engineering startedAt > preprocess finishedAt; 1.0h Argo artifact output/input migration: preprocess declares artifact output, feature-engineering declares artifact input; topology enforcement made automatic).

In a HourTab retainer log, the incident session reads: “Argo Workflows DAG missing dependency investigation: Argo Workflows 3.5.4, Python 3.11, EKS 1.28; daily-ml-feature-pipeline DAGTemplate; feature-engineering step: dependencies: [data-ingestion] (should be [preprocess]); preprocess writes /mnt/pipeline/features.parquet (30–45 min, 18–26 GB); feature-engineering starts concurrently at T+2:43 (29 seconds after preprocess); reads partially written Parquet at T+4:43; ArrowInvalid: Parquet magic bytes not found in footer; retryStrategy: limit: 5, backoff: 5m; retry window 27 min; pipeline fails when preprocess >27 min (6 of 21 days); fix: dependencies: [preprocess]; runtime normalizes 48 min; 4.0h.”

HourTab turns a time-tracker CSV into a public retainer-hours URL your client can bookmark — no login, no portal, no “how many hours do I have left?” emails. Free for one retainer. $9/mo for up to ten.

FAQ: Argo Workflows developer retainers

See also: Kubeflow developer retainer · Prefect developer retainer · ZenML developer retainer