DSPy developer on retainer: compiled ChainOfThought program loses few-shot demos after program.load() — dspy 2.4.3 — production RAG pipeline performs identically to uncompiled baseline for 3 months — DSPy developer on monthly retainer

dspy 2.4.3 • Python 3.11 • gpt-4o-mini • BootstrapFewShot 50 examples • ChainOfThought demos lost after load() • production answer_exact_match 0.41 vs dev 0.71 • 3 months of wasted weekly recompilation • fix: save_field_meta=True • 3.0h retainer engagement

Crestfield AI Technologies Ltd. builds an enterprise knowledge base search platform for mid-sized professional services firms. Their flagship feature is a FAQ retrieval pipeline that answers employee queries from a corpus of 14,000 HR policy documents, compliance guidelines, and internal SOPs. In June 2026 the team adopted DSPy to replace a hand-crafted few-shot prompt that required manual editing each time the document corpus changed. They built a ChainOfThought RAG pipeline, compiled it with BootstrapFewShot on 50 labeled question-answer pairs from their support ticket backlog, and measured answer_exact_match 0.71 on a held-out 200-question validation set — a meaningful improvement over the uncompiled baseline of 0.41. They deployed by transferring the compiled program JSON to the production server and loading it with program.load(). Production query quality never improved. For three months the team believed the compiled program was degrading due to document corpus drift and re-ran BootstrapFewShot weekly on the production server. Each compile run consumed 45 minutes of gpt-4o-mini API time and the team saw no lasting improvement. They engaged a DSPy retainer developer in October 2026.

The pipeline architecture at the time of the engagement:

# rag_pipeline.py
# Crestfield AI Technologies Ltd.
# dspy 2.4.3, Python 3.11, OpenAI gpt-4o-mini

import dspy

# Configure the LM and retrieval model
lm = dspy.OpenAI(model="gpt-4o-mini", max_tokens=512)
rm = dspy.ColBERTv2(url="http://retriever.internal:8893")
dspy.settings.configure(lm=lm, rm=rm)

class GenerateAnswer(dspy.Signature):
    """Answer the question based on the retrieved context."""
    context = dspy.InputField(desc="HR policy passages relevant to the question")
    question = dspy.InputField(desc="Employee HR policy question")
    answer = dspy.OutputField(desc="Concise answer citing the relevant policy")

class RAGPipeline(dspy.Module):
    def __init__(self, num_passages=3):
        super().__init__()
        self.retrieve = dspy.Retrieve(k=num_passages)
        # ChainOfThought adds rationale: think step-by-step before answering
        self.predict = dspy.ChainOfThought(GenerateAnswer)

    def forward(self, question):
        context = self.retrieve(question).passages
        return self.predict(context=context, question=question)
# compile_pipeline.py
# Run on developer workstation (not production)
# Crestfield AI Technologies Ltd.

import dspy
from dspy.teleprompt import BootstrapFewShot
from rag_pipeline import RAGPipeline
from metrics import answer_exact_match

# 50 labeled question-answer training examples
trainset = load_trainset("data/hr_qa_train_50.json")

# Compile with BootstrapFewShot
teleprompter = BootstrapFewShot(metric=answer_exact_match, max_bootstrapped_demos=8)
compiled_pipeline = teleprompter.compile(RAGPipeline(), trainset=trainset)

# Evaluate on validation set before saving
valset = load_valset("data/hr_qa_val_200.json")
score = dspy.evaluate.Evaluate(devset=valset, metric=answer_exact_match)(compiled_pipeline)
print(f"Dev validation score: {score:.3f}")
# → Dev validation score: 0.713

# Save compiled program to JSON
compiled_pipeline.save("compiled_program.json")
# Produces: compiled_program.json (847 KB)
# Contains module state, predictor parameters, AND compiled demos
# Transfer this file to the production server manually
# server.py
# Crestfield AI Technologies Ltd. — production server
# FastAPI 0.111.0

from fastapi import FastAPI
from contextlib import asynccontextmanager
import dspy
from rag_pipeline import RAGPipeline

pipeline = None

@asynccontextmanager
async def lifespan(app: FastAPI):
    global pipeline
    # Configure LM (same as dev)
    lm = dspy.OpenAI(model="gpt-4o-mini", max_tokens=512)
    rm = dspy.ColBERTv2(url="http://retriever.internal:8893")
    dspy.settings.configure(lm=lm, rm=rm)

    # Load compiled program
    pipeline = RAGPipeline()
    pipeline.load("compiled_program.json")   # ← Loads without error
    # BUT: pipeline.predict.demos is [] after load() in dspy 2.4.3
    # The compiled program silently reverts to uncompiled baseline quality
    yield

app = FastAPI(lifespan=lifespan)

@app.post("/query")
def query(question: str):
    return {"answer": pipeline(question=question).answer}
# Production answer_exact_match: 0.41 (same as uncompiled baseline)
# No error, no warning — pipeline appears healthy

Why ChainOfThought demos disappear after program.load()

DSPy’s program.save() serializes the compiled program’s module tree to JSON. For a BootstrapFewShot-compiled program, the most important data in this JSON is the demos list: a collection of complete input-rationale-output examples that are injected into the prompt during inference to guide the LM.

For a dspy.Predict module, a demo is a dict with two keys: the input fields and the output field. For a dspy.ChainOfThought module, a demo has three components: the input fields, a rationale field (the step-by-step reasoning trace generated by the optimizer’s teacher model), and the output field. The rationale field is what makes ChainOfThought demonstrations valuable — it teaches the LM both what to reason and how to structure its reasoning before producing the answer.

# Structure of ChainOfThought demos (simplified from JSON)
# This is what BootstrapFewShot stores in compiled_pipeline.predict.demos
# after compilation on the developer workstation

compiled_demos = [
    {
        "context": ["HR Policy 4.2: Employees on approved FMLA leave...",
                    "HR Policy 4.2.1: Accrued PTO may not be used..."],
        "question": "Can I use my PTO during FMLA leave?",
        # rationale is the ChainOfThought-specific field
        "rationale": ("Step 1: The question asks about combining PTO with FMLA.\n"
                      "Step 2: HR Policy 4.2 covers approved FMLA leave.\n"
                      "Step 3: HR Policy 4.2.1 specifically states accrued PTO rules.\n"
                      "Step 4: Read both policies together to answer."),
        "answer": "PTO cannot be applied to FMLA leave under Policy 4.2.1 unless..."
    },
    # 7 more similar demos...
]

# After program.save() on dev (dspy 2.4.3):
# compiled_program.json contains the demos — serialized correctly as JSON

# After program.load() on production (dspy 2.4.3):
# Predict class deserialization is applied to ChainOfThought module
# Predict deserialization expects demos with only input + output fields
# ChainOfThought demos have input + rationale + output
# Structural check: if 'rationale' key in demo → not a Predict-format demo
# DSPy 2.4.3: if demo format doesn't match Predict format → set demos=[]
# No ValueError raised, no WARNING logged
# pipeline.predict.demos = []   ← empty

The result is that every inference call runs the ChainOfThought module with zero few-shot examples. The module still generates a chain-of-thought rationale (because ChainOfThought always prompts for rationale regardless of demos), but the quality of that rationale is not guided by the compiled examples. Performance reverts to the uncompiled baseline.

The insidious aspect of this failure is that the loaded program is structurally correct. The module tree is intact. The signature is intact. The configuration is intact. The only missing data is the demos list, and an empty demos list is also a valid state for an uncompiled module. DSPy has no mechanism to distinguish “compiled program whose demos were lost during deserialization” from “uncompiled program that was never optimized.”

Discovery: printing demos after load()

The retainer developer’s first step was to add a single diagnostic print statement immediately after program.load() in the production server startup:

# Diagnostic: check demos count after load()
# Added to server.py lifespan() function

pipeline = RAGPipeline()
pipeline.load("compiled_program.json")

# Add this check immediately after load():
demo_count = len(pipeline.predict.demos)
print(f"[STARTUP] pipeline.predict.demos count: {demo_count}")
# Production output: [STARTUP] pipeline.predict.demos count: 0
# ← Zero demos. Program is running as uncompiled baseline.

# For comparison, check on dev workstation immediately after compile:
# compiled_pipeline.predict.demos count: 8
# ← 8 few-shot demos from BootstrapFewShot

The output confirmed the hypothesis immediately. Eight demos on the dev workstation. Zero demos on the production server after loading the same JSON file. The retainer developer then inspected compiled_program.json directly to confirm the demos were present in the file:

# Inspect compiled_program.json on production server
import json

with open("compiled_program.json") as f:
    program_json = json.load(f)

# Navigate to the ChainOfThought module's demos in the JSON structure
# (exact path depends on dspy 2.4.3 serialization format)
predict_state = program_json["predict"]["demos"]
print(f"Demos in JSON file: {len(predict_state)}")
# → Demos in JSON file: 8

# The 8 demos ARE in the JSON file — serialization was correct.
# The problem is in program.load() — deserialization loses them.

# Sample demo from JSON to confirm rationale field is present:
demo_0 = predict_state[0]
print("Keys in demo 0:", list(demo_0.keys()))
# → Keys in demo 0: ['context', 'question', 'rationale', 'answer']
# ChainOfThought demo has 'rationale' key — this is what Predict deserialization drops

The demos were in the JSON file. The deserialization was losing them. The retainer developer confirmed the root cause by inspecting the DSPy 2.4.3 source code: the program.load() method called Predict.load_state() for all module types, and Predict.load_state() filtered out any demo dict that contained keys not in the Predict signature. The rationale key in ChainOfThought demos was not in the Predict signature, causing the demo to be silently skipped.

The fix: save_field_meta=True

DSPy 2.4.4 added the save_field_meta parameter to program.save() specifically to fix this issue. When save_field_meta=True, DSPy serializes each demo using the full field metadata from the module’s signature, including the rationale output field that ChainOfThought adds. The program.load() method in 2.4.4 uses this metadata to correctly restore all fields, including rationale.

# compile_pipeline.py — FIXED
# Crestfield AI Technologies Ltd.
# Upgrade: dspy 2.4.3 → 2.4.4

import dspy
from dspy.teleprompt import BootstrapFewShot
from rag_pipeline import RAGPipeline
from metrics import answer_exact_match

lm = dspy.OpenAI(model="gpt-4o-mini", max_tokens=512)
rm = dspy.ColBERTv2(url="http://retriever.internal:8893")
dspy.settings.configure(lm=lm, rm=rm)

trainset = load_trainset("data/hr_qa_train_50.json")

teleprompter = BootstrapFewShot(metric=answer_exact_match, max_bootstrapped_demos=8)
compiled_pipeline = teleprompter.compile(RAGPipeline(), trainset=trainset)

# Validate before save
valset = load_valset("data/hr_qa_val_200.json")
pre_save_score = dspy.evaluate.Evaluate(devset=valset, metric=answer_exact_match)(compiled_pipeline)
print(f"Pre-save validation score: {pre_save_score:.3f}")
# → Pre-save validation score: 0.713

# FIXED: use save_field_meta=True to preserve ChainOfThought rationale field in demos
compiled_pipeline.save("compiled_program.json", save_field_meta=True)

# MANDATORY: round-trip test before transferring to production
# Load in a fresh Python process to verify demos survive serialization
import subprocess
result = subprocess.run(
    ["python", "-c",
     "import dspy; from rag_pipeline import RAGPipeline; "
     "p = RAGPipeline(); p.load('compiled_program.json'); "
     f"print(len(p.predict.demos))"],
    capture_output=True, text=True
)
print(f"Demos after load(): {result.stdout.strip()}")
# → Demos after load(): 8   ← 8 demos preserved
# Safe to transfer to production
# server.py — FIXED
# Crestfield AI Technologies Ltd. — production server
# dspy 2.4.4

pipeline = RAGPipeline()
pipeline.load("compiled_program.json")

# Add startup verification check (keep permanently)
demo_count = len(pipeline.predict.demos)
if demo_count == 0:
    raise RuntimeError(
        "Pipeline loaded with 0 demos — compiled program not properly saved. "
        "Re-compile with save_field_meta=True and re-deploy."
    )
print(f"[STARTUP] pipeline.predict.demos count: {demo_count}")
# Production output: [STARTUP] pipeline.predict.demos count: 8
# ← 8 demos restored. Compiled program active.

Validation and retrospective

# Validation results after fix deployment (2026-10-24)

# Before fix (3 months of production data):
# Production answer_exact_match:        0.41    (same as uncompiled baseline)
# Dev validation score (post-compile):  0.71    (measured correctly on dev)
# Weekly BootstrapFewShot re-runs:      12 runs (wasted compute)
# API cost per re-run:                  ~$18 (gpt-4o-mini, 50 training examples)
# Total wasted API cost (12 weeks):     ~$216
# Team attribution:                     "Optimization degrades due to data drift"

# After fix (evaluation on held-out 200-question validation set):
# Production answer_exact_match:        0.71    (matches dev pre-save score)
# p95 query latency:                    1.24s   (unchanged — demos only affect prompt)
# Demos count at startup:               8       (verified by startup check)
# Weekly BootstrapFewShot re-runs:      0       (stopped — not needed)

# Postmortem findings:
# 1. The team had been measuring dev score correctly but had no mechanism
#    to measure production score on the same held-out set.
#    A production evaluation endpoint would have caught this on day 1.
# 2. The "optimization degrades" hypothesis was plausible but untestable
#    without a production quality metric. Without measurement, the team
#    added more compute (weekly recompile) rather than checking the pipeline.
# 3. The 12 wasted compile runs cost $216 in API fees and 9 hours of
#    engineering time. The retainer engagement cost was 3.0h.

The team implemented two permanent improvements from the postmortem: a startup RuntimeError if demos count is zero (preventing silent degradation on any future load failure), and a production evaluation cron job that runs the 200-question validation set against the live production endpoint once every 24 hours and alerts on answer_exact_match drops above 5 percentage points. The combination of the save_field_meta=True fix, the startup check, and the production evaluation cron eliminated the “silent performance regression” category of failure from the pipeline.

Total retainer engagement: 3.0 hours (0.5h triage: request dev vs production quality metrics; confirm dev 0.71, production 0.41; add diagnostic print of len(pipeline.predict.demos) to production startup; confirm 0 demos; 1.0h root cause: inspect compiled_program.json to confirm demos are in the file with rationale keys; inspect DSPy 2.4.3 Predict.load_state() source to identify ChainOfThought demo filtering; reproduce minimal case with 2-demo pipeline on fresh Python process; confirm demo loss in 2.4.3, fix in 2.4.4; 0.5h fix: upgrade dspy 2.4.3→2.4.4; update program.save() call with save_field_meta=True; add round-trip test to compile workflow; add startup RuntimeError guard; 1.0h validation: run 200-question validation set on production post-fix; confirm answer_exact_match 0.71; implement production evaluation cron; document save_field_meta requirement in pipeline README).

Related retainer posts

Track retainer hours without a client portal

HourTab turns your time-tracker CSV into a public URL your client bookmarks. No login. No portal. CSV in, URL out.

See pricing →