Fiddler developer on retainer: asyncio fire-and-forget drops 98% of publish_event() calls at 8,000 RPS — drift alert fires on survivor-biased 2% sample, not real feature drift — Fiddler developer on monthly retainer

fiddler-client 2.1.0 • FastAPI 0.104.1 • Python 3.11 • asyncio BackgroundTasks • 8,000 RPS flash sale • task queue saturation • 98% event loss • survivor bias: 2% skewed to high-value slow-path transactions • cart_value PSI 0.31 false drift alert • fix: Redis LPUSH + dedicated fiddler-worker • 3.5h retainer engagement

On October 18, 2026, at 13:47 UTC, the Fiddler model monitoring dashboard for Brightdale ML Monitoring Inc. fired a drift alert for the cart_value feature of their e-commerce churn prediction model. The PSI score was 0.31, well above the configured threshold of 0.20. The ML engineering team immediately began investigating whether a genuine distributional shift had occurred — a new marketing campaign, a product catalog change, or a data pipeline issue upstream of the prediction service. Four hours later, the Fiddler retainer developer reached a different conclusion: the drift signal was an artifact of asyncio task queue saturation that had caused 98% of Fiddler events to be silently dropped during a flash sale, leaving only a survivor-biased 2% sample that made the drift monitor fire on measurement noise, not real drift. The retainer developer closed the incident by 17:30 UTC and delivered the architectural fix by end of day.

Brightdale ML Monitoring uses Fiddler (fiddler-client 2.1.0) to monitor a tabular churn prediction model serving an e-commerce platform. The model takes fourteen features — including cart_value, session_duration_s, premium_member, and days_since_last_purchase — and returns a churn probability. Predictions run through a FastAPI 0.104.1 service deployed on GKE 1.28. The Fiddler integration was added three months earlier: each prediction call appended a background task that published the prediction event to Fiddler:

# prediction_service/main.py — original pattern (pre-incident)
# Brightdale ML Monitoring Inc., fiddler-client 2.1.0, FastAPI 0.104.1

import fiddler as fdl
from fastapi import FastAPI, BackgroundTasks
from pydantic import BaseModel

app = FastAPI()
fiddler_client = fdl.FiddlerApi(url=FIDDLER_URL, org_id=ORG_ID, auth_token=AUTH_TOKEN)

class PredictionRequest(BaseModel):
    cart_value: float
    session_duration_s: int
    premium_member: bool
    days_since_last_purchase: int
    # ... 10 more features

@app.post("/predict")
async def predict(request: PredictionRequest, background_tasks: BackgroundTasks):
    features = request.model_dump()
    prediction = model.predict_proba([list(features.values())])[0][1]

    # Fire-and-forget: enqueues publish_event() as asyncio background task.
    # Appears safe at low throughput. Saturates event loop queue above ~1,200 RPS.
    background_tasks.add_task(
        fiddler_client.publish_event,
        project_id="brightdale-ecom",
        model_id="churn-v4",
        event={**features, "prediction": float(prediction)},
        event_timestamp=int(time.time() * 1000),
    )

    return {"churn_probability": float(prediction)}

At normal load — approximately 800 requests per second during business hours — this integration worked correctly. Fiddler received close to 100% of prediction events. The background tasks had enough event-loop idle time between request yields to execute before new requests arrived. The drift dashboard showed stable PSI scores for all fourteen features over the preceding three months.

On October 18, 2026, Brightdale’s e-commerce partner ran an unannounced flash sale from 13:00 to 17:00 UTC. Prediction traffic peaked at 8,000 requests per second — ten times the normal rate. FastAPI runs on a single asyncio event loop thread per worker process. BackgroundTasks.add_task() appends coroutines to the event loop’s pending task queue. Under normal load, the event loop has idle cycles to drain this queue between incoming requests. At 8,000 RPS, the event loop was fully saturated processing incoming HTTP connections and running prediction inference. The background task queue filled at approximately 1,200 RPS — above that rate, the asyncio scheduler began silently dropping enqueued coroutines that could not be scheduled before the queue memory limit was reached. No exception was raised. No HTTP error was returned to the client. No log line was written by fiddler-client. The prediction response was returned normally. The publish_event() call was simply never executed.

Over the four-hour flash sale, the prediction service handled 800,000 prediction requests. Fiddler received 16,000 events — 2% of the actual volume. The 16,000 events that survived were not a random sample. They were the events whose background tasks happened to be scheduled during the brief pauses between request batches — disproportionately the requests that triggered longer code paths in the prediction handler: premium member authentication checks, database lookups for high-value cart sessions, and edge-case feature encoding for new product categories. These slow-path requests shared a systematic feature bias: cart_value averaged $342 in the surviving 2% versus $78 in the full traffic distribution, and premium_member was true in 61% of surviving events versus 23% of all requests. The Fiddler drift monitor computed PSI between this biased 2% survivor window and the baseline dataset. cart_value PSI came out at 0.31. The drift alert fired at 13:47 UTC.

The retainer developer confirmed the root cause by comparing Fiddler’s ingested event count (16,000 events, visible in Fiddler’s monitoring dashboard as “total events in window”) against the prediction service’s Prometheus request counter (800,000 requests, confirmed via Grafana). The 98% discrepancy was unambiguous. The asyncio saturation hypothesis was confirmed by adding a queue depth counter to the request handler: under simulated 8,000-RPS load in staging, the queue depth grew to the asyncio scheduler’s internal limit within 4 seconds and held there, with subsequent add_task() calls returning immediately without scheduling anything. The fix replaced the fire-and-forget pattern with a durable Redis queue decoupled from the request hot path:

# prediction_service/main.py — fixed pattern
# Brightdale ML Monitoring Inc., fiddler-client 2.1.0, FastAPI 0.104.1
import redis
import json
import time

redis_client = redis.Redis(host=REDIS_HOST, port=6379, db=0)
FIDDLER_QUEUE_KEY = "fiddler:events"

@app.post("/predict")
async def predict(request: PredictionRequest):
    features = request.model_dump()
    prediction = model.predict_proba([list(features.values())])[0][1]

    # LPUSH to Redis is synchronous, non-blocking, and completes in <1ms.
    # No asyncio event loop involvement — safe at any throughput.
    event_payload = json.dumps({
        **features,
        "prediction": float(prediction),
        "event_timestamp": int(time.time() * 1000),
    })
    redis_client.lpush(FIDDLER_QUEUE_KEY, event_payload)

    return {"churn_probability": float(prediction)}

# fiddler_worker/worker.py — dedicated Kubernetes Deployment (1-8 replicas)
# Scales on Redis queue depth via KEDA ScaledObject
import fiddler as fdl
import redis, json, time

fiddler_client = fdl.FiddlerApi(url=FIDDLER_URL, org_id=ORG_ID, auth_token=AUTH_TOKEN)
redis_client = redis.Redis(host=REDIS_HOST, port=6379, db=0)

while True:
    item = redis_client.rpop(FIDDLER_QUEUE_KEY)
    if item is None:
        time.sleep(0.01)
        continue
    event = json.loads(item)
    fiddler_client.publish_event(
        project_id="brightdale-ecom",
        model_id="churn-v4",
        event=event,
        event_timestamp=event.pop("event_timestamp"),
    )

The worker Deployment was configured with a KEDA ScaledObject that autoscaled replica count from 1 to 8 based on Redis queue depth, with a scale-up trigger at queue depth above 1,000. At peak flash-sale throughput in the post-fix stress test, the queue peaked at 4,200 items before the KEDA autoscaler provisioned four additional worker replicas; the queue drained to zero within 90 seconds. Event delivery rate in the post-fix stress test was 99.7% (the 0.3% loss was attributable to a single worker pod restart during scaling, with events in that pod’s batch retried from a dead-letter queue). The retainer developer also added a minimum sample-size guard to all Fiddler drift monitors: alerts are suppressed when the event window contains fewer than 800 events, preventing the drift monitor from firing on statistically unreliable small samples during future low-traffic windows. In the four weeks following the fix, the drift dashboard showed stable PSI scores across all fourteen features, including through two subsequent flash sale events.

Total retainer engagement: 3.5 hours (1.0h root cause: Fiddler event count anomaly detected, asyncio BackgroundTasks saturation hypothesis formed, queue saturation confirmed via staging load test; 0.5h impact assessment: false-positive drift alert classification, event loss quantification from Prometheus vs Fiddler event count, survivor-bias feature distribution analysis; 1.5h fix: Redis LPUSH integration in request handler, fiddler-worker Kubernetes Deployment spec, KEDA ScaledObject configuration, dead-letter queue for failed publish_event() retries; 0.5h post-incident: drift alert validation on representative post-fix events, minimum sample-size guard configuration on all drift monitors, team documentation of asyncio event-publishing prohibition). Retainer billing was logged to HourTab with the fiddler-client version, the asyncio queue saturation mechanism, and the complete Redis worker architecture, so the client could audit the full 3.5-hour work log at the shared URL without logging in.

Related retainer posts

• MLflow developer on retainer — experiment tracking and model registry retainer work
• Evidently developer on retainer — DataDriftPreset small-window false positives retainer work
• Arize developer on retainer — prediction ID collision corrupts delayed-actual joins retainer work