DVC developer on retainer: new engineer runs dvc pull --all-commits and downloads 183 GB of artifact history — 200 GB CI EBS volume exhausted — 6-hour pipeline outage — DVC developer on monthly retainer

DVC 3.48.4 • Python 3.11 • AWS S3 remote • AWS EKS 1.28 • dvc pull --all-commits • full artifact history download • 183 GB / 312 GB total • 200 GB EBS volume exhausted • “No space left on device” • 6-hour CI outage • fix: dvc pull + pre-flight disk check + dvc gc schedule • 3.0h retainer engagement

On October 15, 2026, at 11:12 UTC, the monitoring system for Thornfield Engineering Ltd.’s GitLab CI runners fired a disk-usage alert: the self-hosted EBS volume attached to their primary ML training runner had reached 97% capacity. Three minutes later, at 11:15 UTC, every queued CI job on that runner failed with the same error: No space left on device. The runner was gone, and with it the entire ML training pipeline for the day. Team lead Sofía Reyes correlated the alert time with the CI audit log and found the cause: at 09:47 UTC, new ML engineer Marcus Webb had SSH’d into the runner to troubleshoot a separate CI job and had run dvc pull --all-commits. Over the next two hours and twenty-five minutes, DVC had silently downloaded 183 GB of artifact history from the team’s S3 remote before exhausting the 200 GB gp3 volume. The DVC retainer developer was engaged the same afternoon and closed the incident in 3.0 hours.

Thornfield Engineering’s ML team of eight uses DVC 3.48.4 to version datasets, model checkpoints, preprocessed feature stores, and evaluation reports. The S3 remote, dvc-artifacts-thornfield-prod, holds three years of experiment history totaling 312 GB across all tracked DVC versions. The CI pipeline is designed for incremental work: each training job calls dvc pull before running, fetching only the artifacts tracked in the current Git commit — typically around 4.2 GB per run. The 200 GB EBS volume was sized to accommodate the Docker image layer cache, build artifacts, and several concurrent DVC pulls of current-commit artifacts. It was never intended to hold the team’s entire three-year artifact history.

Marcus Webb was in his first week on the team. While troubleshooting an unrelated CI failure, he SSHed into the runner to inspect the environment. He opened the onboarding documentation at docs/onboarding.md to find the correct dvc pull invocation. The README, written in 2023 when the project was new and the artifact history was small, contained the following code block under the heading “Get all experiment artifacts”:

# docs/onboarding.md — code block as written in 2023, never reviewed
# Thornfield Engineering Ltd., DVC 3.48.4, Python 3.11
# Under heading: "Get all experiment artifacts"

# Fetch current-commit artifacts only (4.2 GB — correct for CI and daily use):
dvc pull

# Get all experiment artifacts (DANGEROUS on large repos — see WARNING below):
# In 2023, this downloaded ~8 GB. In 2026, this downloads ~312 GB.
# DO NOT run on CI runners. Only use for disaster recovery.
dvc pull --all-commits

The README as Marcus read it in October 2026 contained no such warning. The warning shown above is what the file looked like after the incident was fixed. In the original 2023 version, the section simply read:

# docs/onboarding.md — original (pre-incident) version, written 2023
# Under heading: "Get all experiment artifacts"

# Pull the latest artifacts:
dvc pull

# Or to get everything:
dvc pull --all-commits

To a new engineer reading this in their first week, the two commands look equivalent in intent. dvc pull is for normal use; dvc pull --all-commits is for when you want everything. Marcus ran the second command. DVC 3.48.4 does not display a warning, a storage size estimate, or a confirmation prompt before beginning a --all-commits download. The download began immediately at 09:47 UTC. DVC iterated through every Git commit with a DVC-tracked change — spanning 1,200+ commits across three years of experiments — and began fetching every referenced artifact version from the S3 remote into the runner’s DVC cache at /home/gitlab-runner/.dvc/cache.

The timeline of the disk exhaustion was captured in the CI monitoring logs. At 10:23 UTC (36 minutes into the download), the EBS volume was at 78% capacity: 156 GB consumed. At 11:12 UTC (85 minutes in), the monitoring alert fired at the 97% threshold: 194 GB consumed. DVC was still downloading. By 11:15 UTC the volume was full at 183 GB of new downloads (plus pre-existing runner data), and the Linux kernel returned ENOSPC on every file write. The GitLab CI agent, the Docker daemon, and every running CI job received the error simultaneously. Three minutes of CI jobs that had been queued to start that morning all failed. The EKS node hosting the runner had to be replaced entirely — a new node was provisioned with a fresh EBS volume, and the runner came back online at 15:51 UTC, six hours after the outage began.

The root cause diagnosis took 0.5 hours. Sofía Reyes had already correlated the disk alert (11:12 UTC) with the audit log showing Marcus’s SSH session starting at 09:47 UTC. The DVC retainer developer confirmed the mechanism: dvc pull --all-commits internally calls dvc fetch --all-commits followed by dvc checkout --all-commits, which iterates git log to enumerate every commit, reads the .dvc tracking files at each commit, resolves the S3 object keys for every referenced artifact, and downloads them in parallel to the local cache. On a repository where dvc pull normally moves 4.2 GB of data, --all-commits moves the entire 312 GB remote. The flag is designed for a specific, narrow use case — restoring a complete DVC repository from cloud storage after a catastrophic local loss — and is inappropriate for any other context, including developer onboarding.

The fix was deployed in four parts on the same day. First, docs/onboarding.md was updated to replace the dangerous code block with an explicit warning:

# docs/onboarding.md — diff applied October 15, 2026
# Thornfield Engineering Ltd., DVC 3.48.4, Python 3.11

 ## Get all experiment artifacts

-# Pull the latest artifacts:
+# Pull artifacts for the current Git commit (~4.2 GB):
 dvc pull

-# Or to get everything:
-dvc pull --all-commits
+# !! WARNING: DO NOT run the following command for onboarding or daily use !!
+#
+# dvc pull --all-commits downloads EVERY artifact version across ALL Git commits.
+# As of October 2026, this is ~312 GB from the S3 remote.
+# It will exhaust disk space on any machine without 350+ GB free.
+#
+# This command is for DISASTER RECOVERY ONLY (full cold-storage restore).
+# For normal use, always run: dvc pull (no flags)
+#
+# If you need artifacts from a specific past experiment, use:
+#   git checkout 
+#   dvc pull

Second, a CI pre-flight disk space check was added to the GitLab CI configuration. Every job that calls any dvc command now begins with a shell check that fails the job immediately if available disk space is below 20 GB, with an error message that identifies the check and directs the engineer to the runbook:

# .gitlab-ci.yml — pre-flight disk check added October 15, 2026
# Thornfield Engineering Ltd., DVC 3.48.4, AWS EKS 1.28

.dvc_preflight: &dvc_preflight
  before_script:
    - |
      # Pre-flight: verify sufficient disk space before any dvc operation.
      # Minimum required: 20 GB (current-commit pull ~4.2 GB + 4x safety margin).
      # If this check fails, the EBS volume may be nearly full.
      # See runbook: docs/runbooks/ci-disk-exhaustion.md
      AVAIL_KB=$(df --output=avail / | tail -1)
      AVAIL_GB=$((AVAIL_KB / 1024 / 1024))
      echo "CI runner disk available: ${AVAIL_GB} GB"
      if [ "${AVAIL_GB}" -lt 20 ]; then
        echo "ERROR: Insufficient disk space for dvc pull."
        echo "Available: ${AVAIL_GB} GB. Required: 20 GB minimum."
        echo "Check for runaway dvc operations or stale Docker layer cache."
        echo "Runbook: docs/runbooks/ci-disk-exhaustion.md"
        exit 1
      fi

train_model:
  <<: *dvc_preflight
  script:
    - dvc pull
    - python train.py

Third, a scheduled dvc gc job was added to prune S3 artifacts older than 90 days, after the team confirmed a retention policy. The job runs on the first of each month and sends its dry-run output to the team Slack channel for review before executing the actual deletion:

# .gitlab-ci.yml — dvc gc schedule added October 15, 2026
# Thornfield Engineering Ltd., DVC 3.48.4

dvc_gc_monthly:
  rules:
    - if: $CI_PIPELINE_SOURCE == "schedule"
  variables:
    GC_DRY_RUN: "true"  # Set to "false" in the scheduled pipeline after Slack approval
  script:
    - |
      echo "Running dvc gc dry-run to identify reclaimable artifacts..."
      # Retain artifacts referenced by any branch or tag; prune everything else.
      # This removes artifact versions from commits that are no longer reachable
      # from any active Git ref (merged experiment branches, abandoned runs, etc.)
      if [ "${GC_DRY_RUN}" = "true" ]; then
        dvc gc --cloud --all-branches --all-tags --dry-run \
          --remote dvc-artifacts-thornfield-prod \
          2>&1 | tee gc_dry_run_output.txt
        echo "Dry-run complete. Review output above before running without --dry-run."
      else
        dvc gc --cloud --all-branches --all-tags \
          --remote dvc-artifacts-thornfield-prod
        echo "GC complete. Check S3 bucket size in AWS console."
      fi

Fourth, the EBS volume monitoring alert threshold was lowered from 97% to 80% to provide a larger intervention window. At 97% on a 200 GB volume, only 6 GB of headroom remains — far too little to abort a runaway download before the volume fills. At 80%, the alert fires with 40 GB remaining, giving the team time to identify the consuming process and kill it before any CI jobs are impacted. The alert configuration change was applied in the EKS node group launch template and verified with a test that monitored the runner volume via CloudWatch.

The impact assessment took 0.5 hours. The CI outage from 11:15 to 15:51 UTC (6 hours) blocked 14 ML engineer-hours of work. Two model training runs that had been queued at the time of the failure — a fine-tuning job for the team’s production intent classifier and a hyperparameter sweep for a new retrieval model — had to be restarted from scratch on the replacement EKS node. No artifacts in the S3 remote were corrupted or deleted: the incident was a disk exhaustion on the CI runner, not a remote storage event. The dvc pull --all-commits command is read-only with respect to the remote; it downloads from S3 but does not modify it.

On November 1, 2026, the scheduled dvc gc --cloud job ran for the first time with dry-run disabled, after the team reviewed the dry-run output and confirmed that no retained artifacts were referenced by active branches or tags. The job removed 187 GB of artifacts — primarily model checkpoints and preprocessed datasets from experiment branches that had been merged and deleted over the past three years. The S3 bucket shrank from 312 GB to 125 GB, reducing the team’s monthly S3 storage cost by approximately 60%.

Total retainer engagement: 3.0 hours (0.5h root cause: audit log correlation between the 11:12 UTC disk alert and Marcus’s 09:47 UTC SSH session; 0.5h impact assessment: which CI jobs failed, which training runs were lost, whether any artifacts were corrupted; 0.5h fix onboarding README: remove --all-commits, add warning box and safe alternative; 0.5h implement CI pre-flight disk check in .gitlab-ci.yml; 0.5h implement dvc gc --cloud schedule with dry-run approval flow; 0.5h post-incident documentation: runbook update at docs/runbooks/ci-disk-exhaustion.md, EBS alert threshold change, incident timeline for team retrospective). Retainer billing was logged to HourTab with the DVC version, the exact command that triggered the incident, and the full fix applied. The client could verify the complete 3.0-hour work log at the shared HourTab URL without logging in.

Related retainer posts

• MLflow developer on retainer — experiment tracking and model registry retainer work
• Kubeflow developer on retainer — ML pipeline platform retainer work
• ZenML developer on retainer — pipeline orchestration retainer work