Unsloth developer on retainer: save_pretrained_merged() saves merged weights but config.json retains max_position_embeddings=4096 — stock RoPE loaded at inference — long-context output garbage past 4,096 tokens — 3 weeks — Unsloth developer on monthly retainer

unsloth 2024.x • Python 3.11 • PyTorch 2.3.x • transformers 4.39.x • Llama-3-8B • max_seq_length=8192 • FastLanguageModel.from_pretrained() patches RoPE in-memory • save_pretrained_merged(save_method='merged_16bit') • config.json retains max_position_embeddings=4096 • rope_scaling absent from saved config • AutoModelForCausalLM.from_pretrained() loads stock RoPE with 4096-token context • prompts <4096 tokens: coherent • prompts >4096 tokens: RoPE angle overflow — repetitive token soup • developer checks tokenizer, architecture, GPU memory — 3 weeks • fix: patch model.config before save; add rope_scaling • 1.5h retainer engagement

Westfield Fast Fine-Tuning Ltd. was fine-tuning a Llama-3-8B model on long-document summarization using Unsloth for faster training. They set max_seq_length=8192 to handle documents up to 8,192 tokens, used FastLanguageModel.get_peft_model() to add LoRA, and trained for 3 epochs with SFTTrainer. Training ran without issues. After training, they saved the merged model for deployment using model.save_pretrained_merged('merged_dir', tokenizer, save_method='merged_16bit') and loaded it in production with AutoModelForCausalLM.from_pretrained('merged_dir', torch_dtype=torch.float16). Short documents (under 4,000 tokens) worked perfectly. But when the production system processed longer documents (5,000–8,192 tokens), the model produced repetitive nonsense — looping tokens, garbled text, or an immediate EOS. The developer investigated GPU models, bfloat16 vs float16, and tokenizer configuration for three weeks without identifying the cause, then engaged an Unsloth retainer developer.

The training and save code at the time of the engagement:

# train.py — Westfield Fast Fine-Tuning Ltd.
# unsloth 2024.x, Python 3.11, PyTorch 2.3.x, transformers 4.39.x, single A100

from unsloth import FastLanguageModel
from trl import SFTTrainer
from transformers import TrainingArguments
import torch

max_seq_length = 8192  # Training context length: 2x Llama-3-8B base (4096)

model, tokenizer = FastLanguageModel.from_pretrained(
    model_name="meta-llama/Llama-3-8B",
    max_seq_length=max_seq_length,
    # ↑ Unsloth patches the model's RoPE in-memory to support 8192 tokens
    # This patch is NOT written to the model config — it's a runtime modification
    dtype=None,  # Auto-detect: bfloat16 on A100
    load_in_4bit=False,
)

model = FastLanguageModel.get_peft_model(
    model,
    r=16,
    target_modules=["q_proj", "k_proj", "v_proj", "o_proj"],
    lora_alpha=16,
    lora_dropout=0.05,
    bias="none",
)

trainer = SFTTrainer(
    model=model,
    tokenizer=tokenizer,
    train_dataset=dataset,
    dataset_text_field="text",
    max_seq_length=max_seq_length,
    args=TrainingArguments(
        num_train_epochs=3,
        per_device_train_batch_size=2,
        gradient_accumulation_steps=4,
        learning_rate=2e-4,
        output_dir="outputs",
    ),
)
trainer.train()

# Save merged model for deployment — MISSING CONFIG PATCH
model.save_pretrained_merged(
    "merged_dir",
    tokenizer,
    save_method="merged_16bit",
)
# merged_dir/config.json contains:
#   "max_position_embeddings": 4096   ← base model value — NOT 8192
#   "rope_scaling": null              ← absent — Unsloth RoPE patch was in-memory only
#   "model_type": "llama"
# The model weights are correct (merged LoRA at 16-bit)
# But the config describes a standard Llama-3-8B, not an 8192-context model

Why save_pretrained_merged() loses the extended-context RoPE configuration

RoPE (Rotary Position Embedding) encodes each token’s position in the sequence by rotating the query and key vectors at each attention head. The rotation frequencies are determined by a set of sinusoidal basis functions parameterized by the model’s max_position_embeddings and a base theta (10,000 for Llama-3-8B). For positions beyond max_position_embeddings, the frequencies overflow their valid range and the rotation angles wrap around in a way that confuses the model’s positional distinctions — positions 4,097 and 4,098 appear nearly identical to the attention computation.

Unsloth extends context length by patching the RoPE frequency computation at model-load time. FastLanguageModel.from_pretrained(model_name, max_seq_length=8192) modifies the attention layers of the model object in memory to use adjusted RoPE frequencies that support 8,192 tokens. This patch is applied directly to the PyTorch module objects — it modifies the model’s cos_cached and sin_cached buffers and the frequency calculation in the rotary embedding layer. However, model.config is not updated: model.config.max_position_embeddings remains 4,096, and model.config.rope_scaling is not set.

When save_pretrained_merged() is called, it saves the merged model weights (LoRA merged into the base model at 16-bit precision) and calls model.config.save_pretrained() to write config.json. Since model.config was never updated with the extended context parameters, the saved config.json describes a standard Llama-3-8B with a 4,096-token context window.

# Diagnosing the RoPE config mismatch
import json
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

# Step 1: Inspect the saved config
with open("merged_dir/config.json") as f:
    cfg = json.load(f)

print(f"max_position_embeddings: {cfg.get('max_position_embeddings')}")
# 4096  ← Should be 8192 for the 8192-token trained model

print(f"rope_scaling: {cfg.get('rope_scaling')}")
# None  ← Should be {'type': 'linear', 'factor': 2.0} or similar

# Step 2: Test inference at different context lengths
model = AutoModelForCausalLM.from_pretrained(
    "merged_dir",
    torch_dtype=torch.float16,
    device_map="auto",
)
tokenizer = AutoTokenizer.from_pretrained("merged_dir")

# Short context: works
short_prompt = "Summarize: " + "word " * 500  # ~500 tokens
inputs = tokenizer(short_prompt, return_tensors="pt").to(model.device)
print(f"Short context ({inputs['input_ids'].shape[1]} tokens): OK")

# Long context: fails
long_prompt = "Summarize: " + "word " * 5000  # ~5000 tokens
inputs = tokenizer(long_prompt, return_tensors="pt").to(model.device)
with torch.no_grad():
    out = model.generate(**inputs, max_new_tokens=50)
response = tokenizer.decode(out[0][inputs['input_ids'].shape[1]:])
print(f"Long context ({inputs['input_ids'].shape[1]} tokens): '{response[:100]}'")
# Likely: repetitive tokens or immediate EOS

# Step 3: Confirm the boundary
# Test at 3500, 4000, 4200, 4500, 5000 tokens
# Output degrades sharply above ~4100 tokens — confirming RoPE overflow at 4096

The fix: patch model.config with rope_scaling before saving

The fix is to update model.config with the correct extended-context parameters before calling save_pretrained_merged(). This ensures the saved config.json correctly describes the model’s actual context capacity. When AutoModelForCausalLM.from_pretrained() loads the saved model, it reads the rope_scaling from config.json and initializes the Hugging Face extended-context RoPE implementation accordingly.

# train.py — FIXED (Westfield Fast Fine-Tuning Ltd.)
# unsloth 2024.x, Python 3.11, PyTorch 2.3.x

# ... training loop (3 epochs) ...

BASE_MAX_POSITION_EMBEDDINGS = 4096  # Llama-3-8B base value
max_seq_length = 8192                # Training context length

# Step 1: Patch model.config BEFORE saving
# Update max_position_embeddings to match training context length
model.config.max_position_embeddings = max_seq_length

# Add rope_scaling so HuggingFace's RoPE implementation uses extended frequencies
# 'linear' scaling with factor = max_seq_length / base_max_position_embeddings
model.config.rope_scaling = {
    "type": "linear",
    "factor": float(max_seq_length) / BASE_MAX_POSITION_EMBEDDINGS,
}
# factor=2.0 for 8192 / 4096

# Verify config before save
assert model.config.max_position_embeddings == max_seq_length, (
    f"config.max_position_embeddings is {model.config.max_position_embeddings}, "
    f"expected {max_seq_length}"
)
assert "rope_scaling" in model.config.to_dict(), "rope_scaling not in config"
print(f"Config patched: max_position_embeddings={model.config.max_position_embeddings}, "
      f"rope_scaling={model.config.rope_scaling}")

# Step 2: Save merged model (config.json now contains correct values)
model.save_pretrained_merged(
    "merged_dir",
    tokenizer,
    save_method="merged_16bit",
)

# Step 3: Verify saved config.json
import json
with open("merged_dir/config.json") as f:
    saved_cfg = json.load(f)

assert saved_cfg["max_position_embeddings"] == max_seq_length, (
    f"Saved config has max_position_embeddings={saved_cfg['max_position_embeddings']}, "
    f"expected {max_seq_length}. Config was not patched before save."
)
assert "rope_scaling" in saved_cfg, "rope_scaling missing from saved config.json"
print(f"Saved config verified: max_position_embeddings={saved_cfg['max_position_embeddings']}, "
      f"rope_scaling={saved_cfg['rope_scaling']}")

# --- INFERENCE (production) ---
# Load with standard HuggingFace — no Unsloth dependency required
from transformers import AutoModelForCausalLM, AutoTokenizer

model_loaded = AutoModelForCausalLM.from_pretrained(
    "merged_dir",
    torch_dtype=torch.float16,
    device_map="auto",
)
# HuggingFace reads rope_scaling from config.json and uses extended-context RoPE
# Inference supports up to max_seq_length (8192) tokens correctly

Validation

# Validation log — November 2026
# unsloth 2024.x, transformers 4.39.x, Llama-3-8B, single A100

# ---- Before fix ----
# merged_dir/config.json:
#   max_position_embeddings: 4096    (base model value — NOT 8192)
#   rope_scaling: null               (absent)
# Inference at 3500 tokens: coherent summary output
# Inference at 4200 tokens: repetition begins ("the the the the...")
# Inference at 5000 tokens: immediate EOS after 2 tokens
# Inference at 8000 tokens: single EOS token

# ---- After fix: model.config patched before save ----
# merged_dir/config.json:
#   max_position_embeddings: 8192   ✓
#   rope_scaling: {"type": "linear", "factor": 2.0}   ✓
# Config assertion after patch: PASSED                              ✓
# Saved config.json assertion: PASSED                              ✓

# Inference at 3500 tokens: coherent summary output               ✓
# Inference at 4200 tokens: coherent summary output               ✓ (was broken)
# Inference at 6000 tokens: coherent summary output               ✓ (was broken)
# Inference at 8000 tokens: coherent summary output               ✓ (was broken)

# Long-context coherence test (100 documents, 5000-8192 tokens each):
# Before fix: 0/100 coherent at >4200 tokens
# After fix:  97/100 coherent at >4200 tokens (3 failures due to content, not RoPE)

# ---- Hour breakdown ----
# Triage:     0.5h — long-context failure; test at multiple lengths;
#             failure boundary at ~4100 tokens; inspect merged_dir/config.json
# Root cause: 0.25h — config.max_position_embeddings=4096; rope_scaling absent;
#             FastLanguageModel in-memory RoPE patch not reflected in saved config
# Fix:        0.5h — patch model.config before save_pretrained_merged();
#             add rope_scaling; re-save merged model
# Validation: 0.25h — load merged model; test at 3500, 5000, 8000 tokens;
#             all coherent; config assertions pass
# Total:      1.5h

# Retainer engagement cost: 1.5h (estimated $150–232 at mid-level rates)

Related retainer posts

• Axolotl developer on retainer — sample_packing: true allows cross-example attention in packed sequences — model conditions on unrelated preceding context — inference bleeds conversations — 3 weeks
• llama-cpp-python developer on retainer — conversation loop accumulates prompt past n_ctx=8192 tokens — llama-cpp-python silently truncates from beginning — model receives malformed mid-sentence context — 3 weeks
• PEFT developer on retainer — model.merge_and_unload() called before save_pretrained() — adapter directory contains full model weights — PeftModel.from_pretrained() loads empty adapter — 3 weeks