PEFT developer on retainer: model.merge_and_unload() called before save_pretrained() — adapter directory contains full model weights — PeftModel.from_pretrained() loads empty adapter — inference identical to untuned base model — 3 weeks — PEFT developer on monthly retainer

peft 0.10.x • Python 3.11 • PyTorch 2.3.x • transformers 4.39.x • Llama-3-8B • LoRA target_modules=['q_proj','v_proj'] • r=16, lora_alpha=16 • 3 epochs fine-tuning • merge_and_unload() removes PeftModel wrapper • save_pretrained() writes full model weights, not adapter weights • adapter_config.json absent from adapter directory • PEFT 0.10.x silently loads empty adapter over base model • all LoRA weights zero-initialized • inference identical to untuned Llama-3-8B • developer checks lora_alpha, r, target_modules, training loss — 3 weeks • fix: save adapter before merge; load merged with AutoModelForCausalLM • 1.5h retainer engagement

Northfield LLM Fine-Tuning Ltd. was fine-tuning a Llama-3-8B model on a domain-specific question-answering dataset using PEFT LoRA. They configured LoRA with target_modules=['q_proj', 'v_proj'], r=16, and lora_alpha=16 across 4 A100 GPUs. Training ran for 3 epochs and loss decreased from 2.1 to 0.84 — clearly the model was learning. After training, they prepared a merged model checkpoint for deployment by calling model.merge_and_unload() to fold the LoRA weights into the base model, then saved the result with model.save_pretrained('adapter_dir'). The save completed without errors and the adapter directory appeared to contain a large model checkpoint. They then deployed to inference by loading with PeftModel.from_pretrained(base_model, 'adapter_dir'). But the deployed model's responses were identical to the untuned Llama-3-8B base model — it had no knowledge from the fine-tuning. The developer spent three weeks checking lora_alpha, r, target_modules, and training loss curves. They engaged a PEFT retainer developer.

The fine-tuning and save code at the time of the engagement:

# train.py — Northfield LLM Fine-Tuning Ltd.
# peft 0.10.x, Python 3.11, PyTorch 2.3.x, transformers 4.39.x, 4-GPU A100

from peft import LoraConfig, get_peft_model, PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer

base_model = AutoModelForCausalLM.from_pretrained(
    "meta-llama/Llama-3-8B",
    torch_dtype=torch.bfloat16,
    device_map="auto",
)

lora_config = LoraConfig(
    target_modules=["q_proj", "v_proj"],
    r=16,
    lora_alpha=16,
    lora_dropout=0.05,
    bias="none",
    task_type="CAUSAL_LM",
)
model = get_peft_model(base_model, lora_config)
model.print_trainable_parameters()
# trainable params: 13,631,488 || all params: 8,043,196,416 || trainable%: 0.17

# ... training loop (3 epochs, loss 2.1 → 0.84) ...

# Save merged model for deployment — WRONG ORDER
merged_model = model.merge_and_unload()  # Returns plain nn.Module (not PeftModel)
# At this point: type(merged_model).__name__ == 'LlamaForCausalLM'
# merged_model has no PeftModel wrapper, no adapter metadata

merged_model.save_pretrained("adapter_dir")  # Saves FULL model weights, not adapter
# adapter_dir/ contains:
#   config.json                  — LlamaConfig (not LoraConfig)
#   model.safetensors            — 13.4 GB full model weights (merged)
#   generation_config.json
# NOTE: adapter_config.json is ABSENT — this is not an adapter directory

Why PeftModel.from_pretrained() silently loads an empty adapter

model.merge_and_unload() merges the LoRA delta weights into the base model parameters in-place: for each target linear layer, it computes W_new = W_0 + (lora_alpha / r) × B @ A and stores the result back into W_0. After this, the LoRA layers are removed and the model is returned as a plain nn.Module with no PEFT wrapper. When save_pretrained() is called on this plain model, it saves a standard Hugging Face model checkpoint with a config.json and model.safetensors containing the full merged weights — no adapter_config.json, no adapter_model.safetensors.

When PeftModel.from_pretrained(base_model, 'adapter_dir') is called with this directory, PEFT 0.10.x looks for adapter_config.json inside adapter_dir. Finding none, it silently treats the directory as a valid adapter source with empty/default configuration. It initializes LoRA layers on the base model per target_modules defaults — but since no adapter weights are loaded, all LoRA matrices A and B remain zero-initialized. With zero LoRA weights, (lora_alpha / r) × B @ A is zero for every layer, and the model output is mathematically identical to the base model. No error is raised.

# Diagnosing the empty adapter load

import os
from peft import PeftModel
import torch

# Step 1: Inspect the adapter directory
print(os.listdir("adapter_dir"))
# ['config.json', 'model.safetensors', 'generation_config.json']
# Expected: ['adapter_config.json', 'adapter_model.safetensors']
# This is a FULL MODEL directory, not an adapter directory

# Step 2: Load and check adapter weights
peft_model = PeftModel.from_pretrained(base_model, "adapter_dir")

# Inspect LoRA weights
lora_weights = {
    n: p for n, p in peft_model.named_parameters()
    if "lora_" in n
}
print(f"LoRA parameter count: {len(lora_weights)}")
# If 0: PeftModel loaded but found no adapter weights at all

for name, param in list(lora_weights.items())[:4]:
    print(f"{name}: norm={param.norm().item():.6f}")
# Expected after correct adapter load: norm ~0.08 (random init for A, trained B)
# Actual after loading from merged-model dir:
#   lora_A.default.weight: norm=0.000000  ← zero-initialized, not loaded
#   lora_B.default.weight: norm=0.000000  ← zero-initialized, not loaded
# Confirms: adapter weights NOT loaded from adapter_dir

# Step 3: Verify model type at save time
# (diagnostic — run after training, before save)
print(type(model).__name__)   # 'PeftModel' — correct, adapter can be saved
merged = model.merge_and_unload()
print(type(merged).__name__)  # 'LlamaForCausalLM' — plain model, no adapter

The fix: save adapter before merge_and_unload(), load merged with AutoModelForCausalLM

The correct workflow separates adapter saving (for later fine-tuning or A/B testing) from merged-model saving (for efficient deployment inference). Always save the adapter while the model is still a PeftModel, then perform the merge for deployment.

# train.py — FIXED (Northfield LLM Fine-Tuning Ltd.)
# peft 0.10.x, Python 3.11, PyTorch 2.3.x

# ... training loop (3 epochs, loss 2.1 → 0.84) ...

# Step 1: Save adapter WHILE model is still PeftModel
assert type(model).__name__ == "PeftModel", (
    f"Expected PeftModel before save_pretrained(), got {type(model).__name__}. "
    "Call save_pretrained() before merge_and_unload()."
)
model.save_pretrained("adapter_dir")
# adapter_dir/ contains:
#   adapter_config.json          — LoraConfig metadata
#   adapter_model.safetensors    — LoRA A and B weight matrices only (~26 MB)
tokenizer.save_pretrained("adapter_dir")

# Verify adapter directory structure
required = ["adapter_config.json", "adapter_model.safetensors"]
for f in required:
    assert os.path.exists(os.path.join("adapter_dir", f)), (
        f"Adapter directory missing {f} — save was performed on a non-PeftModel"
    )
print("Adapter saved correctly:", os.listdir("adapter_dir"))
# ['adapter_config.json', 'adapter_model.safetensors', 'tokenizer.json', ...]

# Step 2: Create merged model for deployment (separate directory)
merged_model = model.merge_and_unload()
merged_model.save_pretrained("merged_model_dir")
tokenizer.save_pretrained("merged_model_dir")
# merged_model_dir/ contains:
#   config.json, model.safetensors (13.4 GB)

# --- INFERENCE ---

# Option A: Load adapter (for fine-tuning, A/B testing)
from peft import PeftModel
base = AutoModelForCausalLM.from_pretrained("meta-llama/Llama-3-8B", ...)
peft_model = PeftModel.from_pretrained(base, "adapter_dir")

# Verify adapter loaded correctly
lora_weights = {n: p for n, p in peft_model.named_parameters() if "lora_" in n}
assert any(p.norm().item() > 0 for p in lora_weights.values()), (
    "All LoRA weights are zero — adapter may not have loaded. "
    "Check that adapter_dir contains adapter_config.json and adapter_model.safetensors."
)
print(f"Adapter loaded: {len(lora_weights)} LoRA parameters, non-zero norms confirmed")

# Option B: Load merged model (for deployment, no PEFT overhead)
from transformers import AutoModelForCausalLM
merged = AutoModelForCausalLM.from_pretrained("merged_model_dir", ...)
# DO NOT use PeftModel.from_pretrained() for a merged model directory

Validation

# Validation log — November 2026
# peft 0.10.x, transformers 4.39.x, Llama-3-8B, 4-GPU A100

# ---- Before fix ----
# Save sequence: merge_and_unload() → save_pretrained('adapter_dir')
# adapter_dir/: config.json, model.safetensors (13.4 GB)
# adapter_dir/adapter_config.json: ABSENT
# PeftModel.from_pretrained(base, 'adapter_dir') — no error raised
# LoRA weight norms: lora_A=0.000000, lora_B=0.000000 (all zero)
# Inference: identical to untuned Llama-3-8B

# ---- After fix: save adapter before merge ----
# Save sequence: save_pretrained('adapter_dir') → merge_and_unload() → save_pretrained('merged_model_dir')
# adapter_dir/: adapter_config.json (0.9 KB), adapter_model.safetensors (26.4 MB)
# Adapter structure assertion: PASSED                                           ✓
# PeftModel.from_pretrained(base, 'adapter_dir') — loads 13,631,488 LoRA params
# LoRA weight norms: lora_A norm=0.0712, lora_B norm=0.0831 (non-zero)         ✓
# Post-load adapter verification: PASSED                                        ✓

# ---- Inference quality comparison ----
# Prompt: "What is the capital of our product domain?"
# Before fix (empty adapter): [Base model generic response — no domain knowledge]
# After fix (loaded adapter):  [Correct domain-specific response from fine-tuning]
# Fine-tuning reflected in inference: ✓

# ---- Hour breakdown ----
# Triage:     0.5h — inference identical to base model; training loss correct;
#             inspect adapter_dir: model.safetensors (13.4 GB), no adapter_config.json
# Root cause: 0.25h — confirm merge_and_unload() called before save_pretrained();
#             type(model) is LlamaForCausalLM at time of save, not PeftModel
# Fix:        0.5h — reorder save/merge; add type assertion + directory validation
# Validation: 0.25h — adapter_dir contains correct files; LoRA norms non-zero;
#             inference reflects fine-tuning
# Total:      1.5h

# Retainer engagement cost: 1.5h (estimated $150–232 at mid-level rates)

Related retainer posts

Track retainer hours without a client portal

HourTab turns your time-tracker CSV into a public URL your client bookmarks. No login. No portal. CSV in, URL out.

See pricing →