Ray Tune developer on retainer: 2.3 GB dataset loaded inside each trial function — 16 parallel workers × 2.3 GB = 36.8 GB RAM — OOM kills after trial 30 — 3 weeks — Ray Tune developer on monthly retainer

ray[tune] 2.9.x • Python 3.11 • EC2 c5.4xlarge (16 vCPUs, 32 GB RAM) • 200-trial HPO of scikit-learn GradientBoostingClassifier • resources_per_trial={'cpu': 1} • df = pd.read_parquet(path) inside trial function at start of every trial • 16 concurrent workers × 2.3 GB dataset = 36.8 GB peak RSS • Linux OOM killer terminates Ray actor processes • ActorDiedError on retry; retries also OOM • developer checks parquet size, single-trial memory, Ray version — 3 weeks • fix: data_ref = ray.put(df) in outer scope • workers call ray.get(data_ref) from shared plasma store • peak RAM 36.8 GB→2.3 GB • OOM errors 0 • 1.5h retainer engagement

Southfield Ray HPO Ltd. ran a 200-trial hyperparameter search using Ray Tune 2.9.x to optimize a scikit-learn GradientBoostingClassifier for a churn prediction task. Their 2.3 GB training dataset lived on an EFS mount shared across the team. The trial function opened and read the parquet file at the start of each trial — a natural pattern borrowed from single-process training scripts. With resources_per_trial={'cpu': 1}, Ray Tune scheduled up to 16 trials in parallel on the 16-core instance. After 30 trials the job began producing ActorDiedError warnings, trials stopped completing, and the remaining 170 trials were stuck in retry loops that OOM-killed immediately. After three weeks of investigation the team engaged a Ray Tune retainer developer.

The trial function and tune.run() configuration at the time of the engagement:

# Southfield Ray HPO Ltd. — Ray Tune HPO configuration
# ray[tune] 2.9.x, Python 3.11, EC2 c5.4xlarge (16 vCPUs, 32 GB RAM)

import ray
from ray import tune
from ray.tune.search.optuna import OptunaSearch
from ray.tune.schedulers import ASHAScheduler
import pandas as pd
from sklearn.ensemble import GradientBoostingClassifier
from sklearn.metrics import roc_auc_score

ray.init()   # auto-detects 16 CPUs, 32 GB RAM

def trial_fn(config: dict) -> None:
    # ← Problem: loads 2.3 GB parquet at start of every trial
    df = pd.read_parquet("/mnt/efs/southfield/churn_train.parquet")
    X = df.drop(columns=["churned"])
    y = df["churned"]

    model = GradientBoostingClassifier(
        n_estimators=config["n_estimators"],
        max_depth=config["max_depth"],
        learning_rate=config["learning_rate"],
        subsample=config["subsample"],
        min_samples_leaf=config["min_samples_leaf"],
    )
    model.fit(X, y)
    score = roc_auc_score(y, model.predict_proba(X)[:, 1])
    tune.report(train_auc=score)

search_space = {
    "n_estimators": tune.randint(50, 500),
    "max_depth": tune.randint(2, 8),
    "learning_rate": tune.loguniform(0.001, 0.3),
    "subsample": tune.uniform(0.5, 1.0),
    "min_samples_leaf": tune.randint(1, 20),
}

result = tune.run(
    trial_fn,
    config=search_space,
    num_samples=200,
    resources_per_trial={"cpu": 1},   # ← 16 trials run in parallel (all 16 CPUs)
    scheduler=ASHAScheduler(metric="train_auc", mode="max"),
    search_alg=OptunaSearch(metric="train_auc", mode="max"),
    max_failures=3,
)

# Observed failure sequence:
# Trials 1-30: complete normally (workers overlap but partial parallelism < 16)
# Trial 31: 16 workers fully parallel; peak RSS = 16 × 2.3 GB = 36.8 GB > 32 GB
# OOM killer terminates 4 Ray actor processes
# Ray logs: RayActorError: The actor died because of an error raised in its creation task
# Retries: each retry re-runs trial_fn() → reads parquet again → OOM again
# After max_failures=3 retries: trial marked FAILED; next trials inherit same failure
# Job stalls; 170 trials never complete

Why per-trial dataset loading multiplies memory by worker count

Ray Tune runs each trial in a separate Ray actor — a Python process managed by the Ray runtime. When a trial function calls pd.read_parquet(path), the actor process allocates a fresh DataFrame in its own heap. With resources_per_trial={'cpu': 1} on a 16-core machine, Ray schedules up to 16 actors in parallel. Each actor independently reads the parquet file and allocates 2.3 GB. At full parallelism, 16 actors are live simultaneously and total heap consumption reaches 16 × 2.3 GB = 36.8 GB — exceeding the c5.4xlarge’s 32 GB. The Linux OOM killer terminates the actors whose combined RSS pushed the node over the limit, Ray interprets the process exit as an actor crash, and retries the failed trials. Each retry re-runs the same trial function, re-reads the parquet file, and OOM-kills again.

The deceptive aspect: a single-trial memory profile looks correct. If the developer runs memory_profiler on trial_fn() in isolation, it shows a peak of 2.3 GB — accurate. The problem is invisible at the single-trial level and only manifests at the parallelism level, which Ray Tune’s trial-level tracing does not expose.

# Diagnostic: check peak memory at parallelism level

import ray

ray.init()

# Check how many concurrent trials Ray will schedule:
resources = ray.cluster_resources()
max_concurrent = int(resources["CPU"])  # 16 on c5.4xlarge
print(f"Max concurrent trials: {max_concurrent}")
# Output: Max concurrent trials: 16

# Check dataset size:
import os
dataset_size_gb = os.path.getsize("/mnt/efs/southfield/churn_train.parquet") / 1e9
print(f"Dataset size: {dataset_size_gb:.1f} GB")
# Output: Dataset size: 2.3 GB

# Predict peak RAM:
peak_ram_gb = max_concurrent * dataset_size_gb
print(f"Predicted peak RAM: {peak_ram_gb:.1f} GB")
# Output: Predicted peak RAM: 36.8 GB

# Available RAM:
import psutil
available_gb = psutil.virtual_memory().total / 1e9
print(f"Available RAM: {available_gb:.1f} GB")
# Output: Available RAM: 32.0 GB

# Conclusion: 36.8 GB > 32.0 GB → OOM will occur at full parallelism

# Real-time check with ray memory during a run:
# ray memory --stats-only
# Output during full parallelism:
# --- Aggregate object store stats across all nodes ---
# Objects referenced in-scope: 16 objects, 36.8 GiB used
# Worker memory:
#   16 workers, RSS avg 2.3 GB, peak 2.3 GB each
# Total worker RSS: 36.8 GB → matches prediction → confirms OOM cause

The fix: ray.put() before tune.run() to share the dataset via the plasma object store

The fix is to load the dataset once in the outer scope before tune.run(), push it to Ray’s shared plasma object store with ray.put(df), and pass the returned ObjectRef handle to each trial function via tune.with_parameters() or a lambda closure. Inside the trial function, ray.get(data_ref) retrieves the DataFrame from the plasma store. For Arrow-compatible types (Pandas DataFrames backed by Arrow memory, NumPy arrays), Ray returns a zero-copy view into shared memory — the data is not copied into the actor’s heap. For non-Arrow types, Ray deserializes once per actor process and caches the result for the lifetime of the actor. Either way, the dataset lives in the shared plasma store once (2.3 GB) rather than in each actor’s heap (16 × 2.3 GB = 36.8 GB).

# Fix: ray.put() before tune.run() — shared plasma object store
# ray[tune] 2.9.x, Southfield Ray HPO Ltd.

import ray
from ray import tune
from ray.tune.search.optuna import OptunaSearch
from ray.tune.schedulers import ASHAScheduler
import pandas as pd
from sklearn.ensemble import GradientBoostingClassifier
from sklearn.metrics import roc_auc_score

ray.init()

# Load dataset ONCE in the outer scope
df = pd.read_parquet("/mnt/efs/southfield/churn_train.parquet")

# Push to shared plasma object store — returns a lightweight ObjectRef handle
data_ref = ray.put(df)                    # ← 2.3 GB stored once in plasma store

del df    # release outer-scope reference — plasma store is the sole owner now

def trial_fn_fixed(config: dict, data_ref=None) -> None:
    # Retrieve from plasma store — zero-copy for Arrow-backed DataFrames
    df = ray.get(data_ref)                # ← shared memory reference, not a copy
    X = df.drop(columns=["churned"])
    y = df["churned"]

    model = GradientBoostingClassifier(
        n_estimators=config["n_estimators"],
        max_depth=config["max_depth"],
        learning_rate=config["learning_rate"],
        subsample=config["subsample"],
        min_samples_leaf=config["min_samples_leaf"],
    )
    model.fit(X, y)
    score = roc_auc_score(y, model.predict_proba(X)[:, 1])
    tune.report(train_auc=score)

search_space = {
    "n_estimators": tune.randint(50, 500),
    "max_depth": tune.randint(2, 8),
    "learning_rate": tune.loguniform(0.001, 0.3),
    "subsample": tune.uniform(0.5, 1.0),
    "min_samples_leaf": tune.randint(1, 20),
}

# Pass data_ref to trial function via tune.with_parameters()
trial_with_data = tune.with_parameters(trial_fn_fixed, data_ref=data_ref)

result = tune.run(
    trial_with_data,
    config=search_space,
    num_samples=200,
    resources_per_trial={"cpu": 1},
    scheduler=ASHAScheduler(metric="train_auc", mode="max"),
    search_alg=OptunaSearch(metric="train_auc", mode="max"),
    max_failures=3,
)

# Memory after fix (at full 16-trial parallelism):
# Plasma store: 2.3 GB (one copy of dataset)
# 16 worker actors: each holds a zero-copy Arrow view → no additional heap allocation
# Total: 2.3 GB (plasma) + 16 × 0.4 GB (model + sklearn overhead) = 8.7 GB
# Node RAM: 32 GB   Peak: 8.7 GB   Headroom: 23.3 GB   OOM kills: 0

# Note: if the DataFrame contains non-Arrow types (Python objects in cells),
# ray.get() will deserialize (not zero-copy), adding ~2.3 GB per actor on first access.
# In that case: reduce max_concurrent_trials or convert columns to Arrow-compatible types.
# Check: df.dtypes — object dtype columns are non-Arrow.
# Fix:   df[col] = df[col].astype("category") or encode before ray.put().

Validation

# Validation log — November 2026
# ray[tune] 2.9.x, Python 3.11, EC2 c5.4xlarge (16 vCPUs, 32 GB RAM)
# 200-trial HPO of scikit-learn GradientBoostingClassifier

# ---- Before fix: df = pd.read_parquet() inside trial_fn ----
# Max concurrent trials: 16 (all 16 CPUs)
# Dataset loaded per trial: 2.3 GB each
# Peak RSS at full parallelism: 36.8 GB (16 × 2.3 GB)
# Node RAM: 32 GB → OOM at full parallelism
# OOM kills start at: trial 31 (first time all 16 slots filled)
# Trials completed: 30 / 200
# Trials failed (OOM): 170 / 200

# ---- After fix: ray.put(df) → data_ref passed via tune.with_parameters() ----
# Plasma store usage: 2.3 GB (single shared copy)
# Worker RSS per actor: 0.4 GB (model + sklearn overhead, no dataset copy)
# Peak total RAM at full parallelism: 2.3 GB + 16 × 0.4 GB = 8.7 GB
# Node RAM: 32 GB   Peak: 8.7 GB   Headroom: 23.3 GB
# OOM kills: 0
# Trials completed: 200 / 200
# Trial success rate: 100%
# Best trial: n_estimators=312, max_depth=5, learning_rate=0.041,
#             subsample=0.78, min_samples_leaf=4 → train_auc=0.891
# Wall time: 38 minutes (200 trials, 16 parallel, ~3.4 min/trial × 200/16)

# ---- Hour breakdown ----
# Triage:     0.5h — OOM kills after trial 30; ray memory --stats-only
#             showed 16 × 2.3 GB RSS; computed peak_memory = 36.8 GB > 32 GB
# Root cause: 0.25h — pd.read_parquet() inside trial_fn; each actor
#             allocates independent 2.3 GB copy; plasma store not used
# Fix:        0.25h — ray.put(df) before tune.run(); tune.with_parameters()
#             passes ObjectRef to trial_fn; ray.get(data_ref) in trial_fn
# Validation: 0.5h — 200-trial run; zero OOM kills; peak RAM 8.7 GB;
#             same best hyperparams as 30-trial partial run; 1.5h total

Related retainer posts

Track retainer hours without a client portal

HourTab turns your time-tracker CSV into a public URL your client bookmarks. No login. No portal. CSV in, URL out.

See pricing →