whylogs developer on retainer: whylogs.log() called inside PySpark @udf writes profile to ephemeral executor /tmp/ — EMR terminates executor instances on job completion — WhyLabs dashboard shows zero profiles for 21 days — whylogs developer on monthly retainer

whylogs 1.3.5 • Python 3.11 • Apache Spark 3.4.1 • AWS EMR 6.12.0 • @udf decorated function • executor local /tmp/ • ephemeral NVMe storage • profiles destroyed on job completion • WhyLabs “No data” 21 days • fix: SparkWhylogs integration + driver-side .sample().toPandas() • 3.0h retainer engagement

On September 22, 2026, senior ML engineer Amara Okonkwo at Brightwater Analytics Group LLC opened the WhyLabs dashboard to review three weeks of feature quality statistics for the company’s fraud detection model. Every feature panel showed the same message: “No data — waiting for profiles.” The daily Spark job had been running without errors since September 1. All EMR job histories showed exit code 0. The whylogs logging calls produced no exceptions. Twenty-one days of batch inference over 4.2 million transaction rows per day had generated zero feature profiles visible to WhyLabs. The whylogs retainer developer was engaged the same afternoon and identified the root cause in under an hour: the whylogs.log() call was inside a PySpark @udf function, which executes on Spark executor instances, not the driver — and the executor’s local /tmp/ directory is destroyed when the EMR job completes.

Brightwater Analytics built their fraud detection pipeline on Apache Spark running on AWS EMR 6.12.0. The pipeline processes 4.2 million transaction rows daily at 02:00 UTC, applies a feature engineering transformation, scores the features through a pre-trained XGBoost model, and writes predictions to S3. In September 2026, the ML engineering team integrated whylogs 1.3.5 to log feature quality statistics to WhyLabs for ongoing monitoring. The integration was assigned to a mid-level ML engineer who was familiar with whylogs in single-process Python environments but had not previously used it in a distributed Spark context. The engineer’s implementation placed the whylogs.log() call inside a PySpark UDF — a decision that looked architecturally reasonable but violated a fundamental constraint of how Spark executes user-defined functions.

# fraud_inference/feature_profiler.py — Brightwater Analytics Group LLC
# whylogs 1.3.5, Python 3.11, Apache Spark 3.4.1, AWS EMR 6.12.0
# BUG: whylogs.log() called inside a PySpark UDF — profile written to executor /tmp/
# Executor /tmp/ is on ephemeral NVMe, destroyed when EMR job completes.

from pyspark.sql.functions import udf, pandas_udf
from pyspark.sql.types import IntegerType
import whylogs
import pandas as pd

@pandas_udf(IntegerType())
def profile_and_score_partition(feature_series: pd.Series) -> pd.Series:
    """
    BUG: This function executes on Spark executor Python worker processes.
    Any file I/O writes to the executor's local filesystem, not the driver's.
    whylogs.log() writes profile to /tmp/whylogs// on the executor.
    The executor is an EC2 instance (e.g. i-0f4a2b3c4d5e6f789).
    When the Spark job completes, EMR terminates the executor instance.
    Executor /tmp/ is destroyed. Driver /tmp/whylogs/ is empty.
    WhyLabs writer on the driver finds no profiles to upload.
    """
    # Convert partition to DataFrame for whylogs
    partition_df = feature_series.to_frame(name="transaction_amount_usd")

    # BUG: This call succeeds and writes a profile — but to the executor's /tmp/
    # Executor log (i-0f4a2b3c4d5e6f789):
    #   [INFO] whylogs: profile written to /tmp/whylogs/20260915/feature_profile.bin
    # Driver log: no corresponding entry. Profile unreachable from driver.
    result = whylogs.log(partition_df)

    # The scoring logic (correct — UDFs are fine for CPU-bound computation)
    scores = partition_df["transaction_amount_usd"].apply(score_transaction)
    return scores

In PySpark, UDFs — including @pandas_udf decorated functions — are serialized by the Spark driver, transmitted over the network to each executor node, and executed inside the executor’s Python worker process. This is the fundamental Spark execution model: the driver (running on the EMR master node) orchestrates the computation, but the actual data processing happens on executor nodes (running on EMR core and task EC2 instances). Each executor runs in isolation: it has its own JVM, its own Python interpreter, its own local filesystem, and its own network interface. Crucially, the executor’s local filesystem — including /tmp/ — is mounted on the instance’s ephemeral NVMe storage. When the Spark application completes and EMR terminates the executor instances, this storage is destroyed. There is no automatic synchronization of executor local files back to the driver.

The failure was invisible to standard monitoring because every layer of the system reported success. From Spark’s perspective, the UDF executed correctly: it processed each partition, applied the scoring function, and returned results without raising an exception. The whylogs library likewise reported success: it received a valid pandas DataFrame, computed the statistical profile, and wrote a .bin file to /tmp/whylogs/ on the executor’s local disk. The executor-side INFO log message [INFO] whylogs: profile written to /tmp/whylogs/20260915/feature_profile.bin was written to the executor’s stderr log, which is accessible only via the Spark UI’s executor log viewer — not aggregated into the main CloudWatch log stream that the ML engineer monitored. The WhyLabs writer, configured on the driver after the Spark job completed, checked the driver’s local /tmp/whylogs/ path, found it empty, and silently skipped the upload. No exception. No warning. No WhyLabs profile.

The discovery came on September 22. Amara Okonkwo was reviewing the pipeline code in preparation for a feature engineering refactor. She noticed the @pandas_udf decorator on the profiling function and recognized immediately that this meant executor-side execution. To confirm, she opened the Spark UI history server entry for the September 15 job, navigated to the executor tab, selected executor i-0f4a2b3c4d5e6f789, and clicked the stderr log link. The log contained the whylogs INFO messages: one profile write per executor per partition, all to /tmp/whylogs/20260915/ on the executor’s local filesystem. The driver log had no corresponding whylogs entries at all. The whylogs retainer developer, brought in that afternoon, confirmed the diagnosis and implemented the fix.

# fraud_inference/feature_profiler.py — fixed September 23, 2026
# Three options implemented. Option 1 (deployed): driver-side sampling.
# Option 2 (documented): SparkWhylogs native integration.
# Option 3 (documented): per-partition S3 write + driver merge.

import whylogs
import pandas as pd
from whylogs.extras.spark import SparkWhylogs  # Option 2

# ──────────────────────────────────────────────
# OPTION 1 (deployed): Driver-side sampling
# Move whylogs.log() entirely out of the UDF.
# After the Spark job completes, sample 1% of the full DataFrame to the driver.
# 4.2M rows × 0.01 = 42,000 rows — sufficient for accurate statistical profiles.
# All file I/O runs on the driver (EMR master node), which persists after job completion.
# ──────────────────────────────────────────────

def profile_features_on_driver(spark_df, dataset_timestamp: str):
    """
    Sample 1% of the full feature DataFrame to the driver and log with whylogs.
    Runs entirely on the Spark driver — no executor filesystem dependency.
    """
    # .sample() is a Spark transformation (runs on executors, returns distributed DF)
    # .toPandas() collects the sampled rows to the driver as a pandas DataFrame
    sample_df = spark_df.sample(fraction=0.01, seed=42).toPandas()

    # whylogs.log() now runs on the driver — writes to driver /tmp/whylogs/
    # Driver is the EMR master node, which persists after the job completes.
    result = whylogs.log(sample_df)

    # Upload to WhyLabs using the configured writer
    result.writer("whylabs").option("dataset_timestamp", dataset_timestamp).write()
    print(f"[INFO] whylogs: profiled {len(sample_df)} rows, uploaded to WhyLabs.")


# ──────────────────────────────────────────────
# OPTION 2 (documented): SparkWhylogs native integration
# whylogs.extras.spark.SparkWhylogs handles executor-to-driver profile merging
# using Spark accumulators. Profiles from all partitions are merged on the driver.
# Use this when you need full-population statistics (not a sample).
# ──────────────────────────────────────────────

def profile_features_spark_native(spark_df, dataset_timestamp: str):
    """
    Use SparkWhylogs integration for full-population profiling.
    SparkWhylogs uses Spark accumulators to merge per-partition profiles on the driver.
    No executor /tmp/ dependency. Compatible with EMR, Databricks, and GCP Dataproc.
    """
    spark_whylogs = SparkWhylogs()
    # profile_view is a DatasetProfileView merged from all executor partitions
    profile_view = spark_whylogs.profile(spark_df)
    result = whylogs.log(profile_view=profile_view)
    result.writer("whylabs").option("dataset_timestamp", dataset_timestamp).write()
    print(f"[INFO] SparkWhylogs: full-population profile uploaded to WhyLabs.")

The fix deployed on September 23 used Option 1 — the driver-side sampling approach — because it required the smallest change to the existing pipeline architecture and could be validated immediately. A 1% sample of 4.2 million rows yields 42,000 rows, which is more than sufficient for whylogs to compute accurate statistical profiles for all 47 monitored features. For continuous numeric features like transaction_amount_usd, whylogs uses a KLL sketch algorithm that produces accurate quantile estimates from samples of this size. For categorical features like merchant_category_code and card_present_flag, 42,000 rows provides robust frequency distribution estimates. The sample is drawn with a fixed seed (seed=42) for reproducibility, so profile comparisons across days are not affected by sampling variance.

WhyLabs received its first profile for the Brightwater pipeline on September 23. Within minutes, the WhyLabs dashboard populated with statistics for all 47 features. The team could see that the features had been stable throughout the preceding period (the model continued to run correctly; it was only the monitoring layer that was dark). The real value of the integration became apparent on October 3, when a genuine data quality alert fired: the P99 of transaction_amount_usd had increased 3.4 times relative to the WhyLabs reference profile, from approximately $2,400 to $8,200. Investigation revealed that a new high-value merchant category (corporate travel booking) had been added to the transaction processing system on October 1 and was generating legitimate high-value transactions. The alert was genuine — the feature distribution had shifted — and the ML team updated the WhyLabs reference profile to include the new merchant category. Without the whylogs integration working correctly, this distribution shift would have been invisible until model performance metrics degraded.

The whylogs retainer developer also delivered a team documentation artifact: a one-page runbook titled “whylogs in PySpark: the executor isolation rule.” The core rule, printed at the top: Never call whylogs.log() inside a UDF, mapInPandas, applyInPandas, foreachPartition, or any other function that executes on a Spark executor. whylogs has no built-in Spark awareness. File writes inside these functions write to the executor’s local ephemeral filesystem, which is destroyed when the job completes. Use SparkWhylogs for full-population profiles, or .sample().toPandas() on the driver for sampled profiles. The document was added to the team’s ML platform onboarding guide and included a code review checklist item: any PR that adds a whylogs.log() call must be reviewed to confirm the call site is not inside a function that could execute on a Spark executor.

Total retainer engagement: 3.0 hours. Breakdown: 0.5h log archaeology (navigate Spark UI history server to executor stderr logs for September 15 job; confirm whylogs INFO messages on executor i-0f4a2b3c4d5e6f789; confirm driver log has no corresponding whylogs entries); 0.5h diagnose UDF execution model (confirm @pandas_udf executes on executor Python worker process; confirm executor /tmp/ is on ephemeral NVMe; confirm driver /tmp/whylogs/ empty after job completion); 1.0h implement and test fix (write driver-side sampling function; test locally with PySpark in local mode; deploy to EMR; verify September 23 job writes profile to driver /tmp/whylogs/); 0.5h validate WhyLabs receives profiles (confirm all 47 features appear in WhyLabs dashboard; verify profile row counts match expected 42,000-row sample; confirm dataset timestamps align with daily 02:00 UTC job schedule); 0.5h document Spark-whylogs integration pattern (write executor isolation rule runbook; add code review checklist item; document SparkWhylogs as alternative for full-population profiling). Retainer billing was logged to HourTab with all component timestamps and the WhyLabs profile confirmation screenshots, giving Brightwater Analytics full visibility into the engagement work log at their shared HourTab URL.

Related retainer posts

• MLflow developer on retainer — experiment tracking and model registry retainer work
• Great Expectations developer on retainer — data quality validation and expectation suite retainer work
• Evidently developer on retainer — ML model monitoring and data drift detection retainer work