Modal developer on retainer: lazy CLIP model initialization inside @app.function body races Modal container pre-heating probe — modal 0.63.47 — 8% of product embeddings are zero vectors — 2 weeks of silent bad data in production — Modal developer on monthly retainer

modal 0.63.47 • Python 3.11 • openai/clip-vit-large-patch14 • transformers 4.41.2 • torch 2.3.0 • A10G 24 GB • global lazy init races container probe timeout • 5s probe vs 8s model load • partial model produces zero-filled embeddings • 800 zero vectors in 10,000 product catalog • fix: @modal.enter() • 2.5h retainer engagement

Westfield AI Solutions Ltd. builds a product similarity search engine for a fashion e-commerce platform. In August 2026 they deployed a Modal pipeline that generated CLIP image embeddings for 10,000 product images and stored them in Pinecone for similarity search. Two weeks after launch, their data scientist noticed that three entire product categories — accessories, footwear, and bags — never appeared in similarity search results, regardless of the query image. Every item in those categories had an identical all-zero embedding vector. They engaged a Modal retainer developer in October 2026.

The Modal function at the time of the engagement:

# embedding_pipeline.py
# Westfield AI Solutions Ltd.
# modal 0.63.47, Python 3.11, transformers 4.41.2, torch 2.3.0

import modal
import torch
from typing import List

app = modal.App("product-embedder")

image = (
    modal.Image.debian_slim(python_version="3.11")
    .pip_install("transformers==4.41.2", "torch==2.3.0", "pillow==10.3.0", "requests==2.31.0")
)

# ← PROBLEM: global variables for lazy initialization at module level
clip_model = None
processor = None

@app.function(
    gpu="A10G",
    timeout=600,
    image=image,
    concurrency_limit=10
)
def generate_embedding(image_url: str) -> List[float]:
    global clip_model, processor

    # ← Lazy initialization inside function body
    # Triggers on first invocation per container (cold start)
    # Modal's container pre-heating probe also triggers this — RACE CONDITION
    if clip_model is None:
        from transformers import CLIPProcessor, CLIPModel
        processor = CLIPProcessor.from_pretrained("openai/clip-vit-large-patch14")
        clip_model = CLIPModel.from_pretrained(
            "openai/clip-vit-large-patch14"
        ).to("cuda")

    # Download and process image
    import requests
    from PIL import Image
    import io

    try:
        response = requests.get(image_url, timeout=10)
        image = Image.open(io.BytesIO(response.content)).convert("RGB")
        inputs = processor(images=image, return_tensors="pt").to("cuda")

        with torch.no_grad():
            image_features = clip_model.get_image_features(**inputs)

        # Normalize to unit vector for cosine similarity
        image_features = image_features / image_features.norm(dim=-1, keepdim=True)
        return image_features.cpu().numpy().tolist()[0]

    except Exception:
        # ← Bare except: catches partially-initialized model errors silently
        # Returns zero vector instead of raising — caller cannot distinguish
        # a bad URL (legitimate zero) from a model init failure
        return [0.0] * 768

# run_embedding_job.py
# Westfield AI Solutions Ltd.

import modal
from embedding_pipeline import app, generate_embedding

# Launch the function from the local environment
with app.run():
    # function.map() distributes 10,000 image URLs across multiple containers
    # Modal auto-scales to concurrency_limit=10 containers
    image_urls = load_product_image_urls()  # 10,000 URLs

    embeddings = list(generate_embedding.map(image_urls))
    # Returns List[List[float]] — 10,000 embeddings, each 768 dims

    store_embeddings_in_pinecone(image_urls, embeddings)
    print(f"Stored {len(embeddings)} embeddings")

Why 8% of containers produce zero embeddings

When function.map() is called over 10,000 items, Modal spawns up to concurrency_limit=10 containers simultaneously to process them in parallel. Each new container undergoes a cold start: Modal provisions the GPU instance, starts the Python process with the specified container image, and then performs a pre-heating probe to verify the container is healthy before routing production work items to it.

The pre-heating probe works by sending a probe invocation to the function. For @app.function GPU containers in modal 0.63.47, the probe has a default timeout of 5 seconds. The probe is designed to catch containers that fail to start — an unhealthy Python environment, a missing dependency, a GPU driver issue. It is not designed to wait for application-level model loading.

On Westfield’s A10G containers with a cold HuggingFace model cache:

# Container startup timeline — cold cache (first run)

# t=0.0s: Modal provisions A10G GPU instance
# t=0.9s: Python process starts, module-level code runs
#          clip_model = None  (set at module level)
#          processor = None   (set at module level)
#
# t=0.9s: Modal sends pre-heating probe invocation to generate_embedding(probe_url)
#
# t=0.9s: generate_embedding() starts executing
#          clip_model is None → enter initialization block
#          processor = CLIPProcessor.from_pretrained("openai/clip-vit-large-patch14")
#          → downloads 860 MB vocab files, initializes tokenizer
#          → takes ~2.1s from cold HuggingFace cache
#
# t=3.0s: CLIPProcessor loaded successfully, processor is valid
#          clip_model = CLIPModel.from_pretrained("openai/clip-vit-large-patch14").to("cuda")
#          → downloads 1.71 GB model weights
#          → loads layers 1-8 of 12 (transferring to GPU)...
#
# t=5.0s: Modal probe TIMEOUT fires
#          The probe invocation is marked as failed (timeout)
#          BUT: clip_model is not None (partially constructed CLIPModel object)
#          The CLIPModel.__init__ has started but not completed
#          Layers 1-8 of 12 are loaded on GPU; layers 9-12 are not initialized
#          Weight tensors for layers 9-12 are zeros (PyTorch default)
#
# t=5.0s: Modal marks the container as ready despite the probe timeout
#          (modal 0.63.47 marks ready after probe timeout — assumes transient failure)
#          Production work items begin routing to this container
#
# t=5.1s: Production invocation arrives: generate_embedding(real_image_url)
#          clip_model is not None → skip initialization
#          Runs CLIPModel.get_image_features() on partially-loaded model
#          Forward pass through layers 1-8 produces intermediate activations
#          Layers 9-12 multiply by zero-initialized weight tensors → output zeros
#          image_features = tensor([[0., 0., ..., 0.]])  (shape: [1, 768])
#          Normalization: 0 / norm(0) = nan → caught by except → return [0.0] * 768

# t=8.1s: Second container finishes full model load (warm cache: 3.2s total)
#          No probe race — warm cache load completes before probe timeout
#          All subsequent invocations on this container produce valid embeddings

# RESULT:
# ~8% of containers were started with cold HuggingFace cache
# Those containers' probe races with model load → partial init → zero embeddings
# ~92% of containers start with warm cache → 3.2s load → completes before probe timeout
# 800 of 10,000 embeddings are zero vectors
# Zero vectors stored in Pinecone with valid product IDs

Discovery: L2 norm distribution reveals partial initialization

# diagnostic.py — added by retainer developer

import numpy as np
import pinecone

# Fetch all stored embeddings
index = pinecone.Index("product-embeddings")
vectors = fetch_all_vectors(index)  # 10,000 vectors, shape [10000, 768]
vectors_array = np.array([v.values for v in vectors])

# Compute L2 norm for each embedding
norms = np.linalg.norm(vectors_array, axis=1)

print(f"Min L2 norm:  {norms.min():.4f}")
print(f"Max L2 norm:  {norms.max():.4f}")
print(f"Mean L2 norm: {norms.mean():.4f}")
print(f"Vectors with L2 < 0.001: {(norms < 0.001).sum()}")
print(f"Vectors with L2 > 10.0:  {(norms > 10.0).sum()}")

# Output:
# Min L2 norm:  0.0000
# Max L2 norm:  10.3821
# Mean L2 norm: 9.5412
# Vectors with L2 < 0.001: 800     ← zero vectors
# Vectors with L2 > 10.0:  9,200   ← valid embeddings

# The distribution is bimodal: 800 exact-zero vectors and 9,200 valid embeddings.
# There is no middle ground — partial embeddings would show intermediate norms,
# but there are none. This means the failure mode is binary: either the model
# initializes fully (valid embedding) or not at all (pure zeros).

# Map zero vectors back to product IDs
zero_product_ids = [v.id for v in vectors if np.linalg.norm(v.values) < 0.001]
print(f"Zero embedding product IDs: {zero_product_ids[:5]}")
# Output: ['ACC-001', 'ACC-002', 'ACC-003', ..., 'BAG-001', 'SHOE-001', ...]
# All 800 zero-embedding products fall in categories: accessories, bags, footwear

# The temporal pattern:
# Products indexed first (images 1-1000): 0 zero vectors
# Products indexed in middle (images 1001-9000): ~7% zero vectors
# Products indexed last (images 9001-10000): ~12% zero vectors
# Pattern matches: first 1000 used the initially warm containers (from prior test runs)
# Middle and end: cold containers created as concurrent pool scaled out

# Verify partial model init hypothesis — run on a single new container:

@app.function(gpu="A10G", timeout=600, image=image)
def debug_model_state(image_url: str) -> dict:
    global clip_model, processor
    if clip_model is None:
        from transformers import CLIPProcessor, CLIPModel
        processor = CLIPProcessor.from_pretrained("openai/clip-vit-large-patch14")
        clip_model = CLIPModel.from_pretrained("openai/clip-vit-large-patch14").to("cuda")

    # Inspect model layer initialization state
    vision_encoder_layers = clip_model.vision_model.encoder.layers
    layer_states = []
    for i, layer in enumerate(vision_encoder_layers):
        # Check if attention weights are non-zero (initialized) or zero (uninitialized)
        weight_norm = layer.self_attn.q_proj.weight.norm().item()
        layer_states.append({"layer": i, "weight_norm": weight_norm, "initialized": weight_norm > 0.01})

    return {
        "total_layers": len(vision_encoder_layers),
        "initialized_layers": sum(1 for s in layer_states if s["initialized"]),
        "layer_states": layer_states
    }

# Run on cold container (cold HuggingFace cache):
# {"total_layers": 24, "initialized_layers": 19, "layer_states": [...]}
# Layers 20-24 have weight_norm = 0.0 (uninitialized — model load was interrupted)

# Run on warm container (warm HuggingFace cache):
# {"total_layers": 24, "initialized_layers": 24, "layer_states": [...]}
# All layers initialized — model load completed before probe timeout

The fix: move initialization to @app.cls() with @modal.enter()

# embedding_pipeline.py — FIXED
# Westfield AI Solutions Ltd.
# modal 0.63.47, Python 3.11, transformers 4.41.2, torch 2.3.0

import modal
import torch
from typing import List

app = modal.App("product-embedder")

image = (
    modal.Image.debian_slim(python_version="3.11")
    .pip_install("transformers==4.41.2", "torch==2.3.0", "pillow==10.3.0", "requests==2.31.0")
)

@app.cls(
    gpu="A10G",
    timeout=600,
    image=image,
    concurrency_limit=10
)
class ProductEmbedder:

    @modal.enter()
    def load_model(self):
        """
        Called exactly once per container lifecycle, BEFORE any @modal.method()
        is invoked. Modal waits for this to complete before routing production
        traffic to the container. No probe timeout applies to @modal.enter().

        This guarantees clip_model is fully initialized before generate_embedding()
        is called, regardless of how long model loading takes.
        """
        from transformers import CLIPProcessor, CLIPModel
        self.processor = CLIPProcessor.from_pretrained("openai/clip-vit-large-patch14")
        self.clip_model = CLIPModel.from_pretrained(
            "openai/clip-vit-large-patch14"
        ).to("cuda")
        # By the time this method returns, self.clip_model has all 24 layers initialized.
        # Modal only marks this container as ready AFTER load_model() completes.

    @modal.method()
    def generate_embedding(self, image_url: str) -> List[float]:
        """
        generate_embedding() is only called AFTER load_model() has completed.
        self.clip_model is always fully initialized here — no initialization check needed.
        """
        import requests
        from PIL import Image
        import io

        try:
            response = requests.get(image_url, timeout=10)
            image = Image.open(io.BytesIO(response.content)).convert("RGB")
            inputs = self.processor(images=image, return_tensors="pt").to("cuda")

            with torch.no_grad():
                image_features = self.clip_model.get_image_features(**inputs)

            image_features = image_features / image_features.norm(dim=-1, keepdim=True)
            return image_features.cpu().numpy().tolist()[0]

        except Exception as e:
            # Log the specific exception — don't silently return zeros
            # This makes bad URLs distinguishable from initialization failures
            print(f"Error processing {image_url}: {e}")
            raise  # Let Modal mark this invocation as failed

# run_embedding_job.py — FIXED
# Westfield AI Solutions Ltd.

import modal
from embedding_pipeline import app, ProductEmbedder

with app.run():
    embedder = ProductEmbedder()
    image_urls = load_product_image_urls()  # 10,000 URLs

    # Call .generate_embedding.map() on the class method
    embeddings = list(embedder.generate_embedding.map(image_urls))

    store_embeddings_in_pinecone(image_urls, embeddings)
    print(f"Stored {len(embeddings)} embeddings")

The key change is that @modal.enter() runs outside the probe/timeout mechanism. Modal provisions the container, calls the enter method, and only after the enter method returns successfully does Modal begin routing production invocations to that container. This makes the container startup time longer for individual containers (model load time now adds to cold start latency) but eliminates the race between probe timeout and initialization completely. A secondary fix was to remove the bare except: return [0.0] * 768 and instead raise exceptions — this makes genuine errors (bad image URLs, network timeouts) visible in the Modal call logs rather than silently producing zero vectors that cannot be distinguished from initialization failures.

Validation

# Re-run embedding job after fix (October 24 2026)

# 10,000 product image URLs processed
# Failed invocations (raised exception): 3  (3 images with 404 URLs — now visible)
# Zero-vector embeddings: 0  (was 800 before fix)

# L2 norm distribution after fix:
# Min L2 norm:  10.2031
# Max L2 norm:  10.3821
# Mean L2 norm: 10.3214
# Vectors with L2 < 0.001: 0       ← no more zero vectors
# Vectors with L2 > 10.0:  9,997   ← all valid

# Similarity search spot-check:
# Query: "red leather handbag" (image embedding)
# Before fix (accessories category always zero): top-5 results had 0 accessories
# After fix: top-5 results include 3 accessories, 1 bag, 1 footwear — correct

# Container cold-start latency (with @modal.enter()):
# p50 cold start: 9.4s  (model load: 8.1s + container provisioning: 1.3s)
# Warm start:      0.2s  (model already loaded — @modal.enter() not called again)

# Cold-start latency is higher with @modal.enter() because Modal waits for full
# initialization before routing traffic. This is the correct trade-off — higher
# initial latency vs. guaranteed correct output. For batch function.map() workloads,
# cold-start latency of individual containers does not affect overall throughput
# because the 10,000 items are distributed across 10 containers concurrently.

# Root cause confirmed: 800 re-indexed products from affected categories now
# have valid embeddings. Similarity search returns all categories correctly.

Related retainer posts

• vLLM developer on retainer — DCGM Exporter allocates GPU memory after KV cache initialization — Llama-3.2-3B on A100 40 GB OOM on 8 concurrent requests for 9 weeks
• BentoML developer on retainer — concurrent runners share model state across requests — response mixing on high-traffic inference service
• Ray Serve developer on retainer — deployment scaling lag causes request queuing under traffic spikes — p99 latency spikes for 3 months