tokenizers developer on retainer: add_special_tokens=True prepends BOS token 128000 — data collator also manually prepends token 128000 — every training sample begins [128000, 128000, …] — fine-tuned Llama-3-8B expects double BOS — inference with single BOS degrades responses — instruction-following F1 0.84→0.61 — 3 weeks — tokenizers developer on monthly retainer

transformers 4.42.x • tokenizers 0.19.x • Python 3.11 • Llama-3-8B instruction fine-tuning • 10,000 instruction-response pairs • tokenizer.encode(text, add_special_tokens=True) prepends BOS 128000 • data collator also prepends [tokenizer.bos_token_id] • every training sample: [128000, 128000, …] • fine-tuned model learns double-BOS pattern • inference: add_special_tokens=True without manual prepend → single BOS [128000, …] • distribution shift • instruction-following F1 0.61 (was 0.84) • developer checks learning rate, data quality, epochs — 3 weeks • fix: remove manual BOS prepend from collator • F1 0.61→0.84 • 1.5h retainer engagement

Westfield Tokenizer Fine-Tuning Ltd. built an instruction fine-tuning pipeline for Llama-3-8B using the HuggingFace transformers and tokenizers libraries. Their dataset contained 10,000 instruction-response pairs formatted as system prompt + instruction + expected response. The developer had previously fine-tuned LLaMA-2 models using a tutorial that used add_special_tokens=False and manually prepended the BOS token in the data collator. When switching to Llama-3, the developer updated the tokenizer and model but carried over the same data collator code from the LLaMA-2 pipeline, including the manual BOS prepend.

The fine-tuning run converged normally—training loss decreased from 2.1 to 1.1 over two hours. The developer evaluated the fine-tuned model against a 200-sample held-out test set using instruction-following F1. The base Llama-3-8B scored 0.63 on the same test set; the fine-tuned model scored 0.61. Fine-tuning had made the model worse. The developer attributed the regression to insufficient training data and tried training with upsampled data (20,000 pairs), different learning rates (1e-5, 2e-5, 5e-5), different epoch counts (1, 2, 3), and different chat template formats. All runs produced fine-tuned models scoring 0.59–0.63 on instruction-following F1—never above the base model’s 0.63. After three weeks the developer engaged a tokenizers retainer developer.

The data preparation pipeline at the time of the engagement:

# Data preparation — Westfield Tokenizer Fine-Tuning Ltd.
# transformers 4.42.x, tokenizers 0.19.x, Python 3.11
# Llama-3-8B instruction fine-tuning on 10,000 instruction-response pairs

from transformers import AutoTokenizer
from torch.utils.data import Dataset

tokenizer = AutoTokenizer.from_pretrained(
    "meta-llama/Meta-Llama-3-8B-Instruct",
    padding_side="right",
)
# Llama-3 tokenizer special tokens:
#   BOS: <|begin_of_text|> = token id 128000
#   EOS: <|end_of_text|>  = token id 128001
# tokenizer.bos_token_id = 128000
# tokenizer.eos_token_id = 128001

class InstructDataset(Dataset):
    def __init__(self, pairs, tokenizer):
        self.samples = []
        for instruction, response in pairs:
            system_prompt = "You are a helpful assistant. Answer concisely."
            formatted = (
                f"<|system|>{system_prompt}<|user|>{instruction}"
                f"<|assistant|>{response}<|end_of_text|>"
            )
            # tokenizer.encode() with add_special_tokens=True:
            # For Llama-3, this prepends BOS token 128000 automatically
            tokenized = tokenizer.encode(
                formatted,
                add_special_tokens=True,   # ← prepends BOS 128000
            )
            # tokenized[0:3] = [128000, <|system|>_token_id, ...]
            # tokenized sequence: [128000, content..., 128001]
            self.samples.append(tokenized)

    def __getitem__(self, idx):
        return self.samples[idx]

def data_collator(batch):
    """
    Collates tokenized samples into a batch for causal LM training.
    """
    input_ids_list = []
    for input_ids in batch:
        # BUG: carried over from LLaMA-2 pipeline where add_special_tokens=False
        # was used and BOS was added here manually.
        # In the Llama-3 pipeline, add_special_tokens=True already prepended BOS.
        # This prepend adds a SECOND BOS token.
        input_ids = [tokenizer.bos_token_id] + input_ids   # ← BUG: 128000 prepended again
        # input_ids[0:3] = [128000, 128000, content...]
        #                   ↑ manual prepend (from LLaMA-2 collator)
        #                           ↑ already-prepended by tokenizer.encode(add_special_tokens=True)
        #
        # Every training sample begins: [128000, 128000, <|system|>token, system_text, ...]
        # 10,000 samples × 10 training steps = model fine-tuned on 100,000 sequences
        # all beginning with two consecutive BOS tokens.
        input_ids_list.append(torch.tensor(input_ids))

    # Pad and return
    padded = torch.nn.utils.rnn.pad_sequence(
        input_ids_list, batch_first=True, padding_value=tokenizer.pad_token_id or 0
    )
    labels = padded.clone()
    labels[padded == (tokenizer.pad_token_id or 0)] = -100   # mask padding from loss
    return {"input_ids": padded, "labels": labels, "attention_mask": (padded != 0).long()}

# --- inference.py (inference pipeline — does NOT have the manual BOS prepend) ---
def run_inference(model, tokenizer, instruction):
    system_prompt = "You are a helpful assistant. Answer concisely."
    formatted = f"<|system|>{system_prompt}<|user|>{instruction}<|assistant|>"
    inputs = tokenizer(
        formatted,
        return_tensors="pt",
        add_special_tokens=True,   # prepends BOS 128000 once
    ).to("cuda")
    # inputs["input_ids"][0][0:3] = [128000, <|system|>_id, system_text_id, ...]
    #   ↑ single BOS (correct for a fresh inference prompt)
    #
    # Training: model saw [128000, 128000, <|system|>_id, ...]
    # Inference: model sees [128000, <|system|>_id, ...]
    #
    # The model's attention at position 0 (token 128000) was always conditioned
    # on another 128000 token at position 1 during training.
    # At inference, position 1 is <|system|>_id — a token the model never saw
    # immediately after a single BOS during training.
    # Degradation: system prompt conditioning is weakened; model partially ignores
    # the system prompt; instruction-following quality drops.
    return model.generate(**inputs, max_new_tokens=256, do_sample=False)

Why double BOS degrades fine-tuned model quality

Llama-3’s BOS token <|begin_of_text|> (id 128000) signals the start of a new document or conversation. During pre-training on trillions of tokens, the model learned that every sequence begins with exactly one BOS token. The model’s internal representations at position 0 and position 1 are conditioned on what immediately follows the BOS.

When fine-tuning places [128000, 128000, content] at the start of every training sample, the model must adapt its behavior at position 0 to predict another BOS token (128000) instead of the system prompt token. The model learns a new conditional distribution: “given position 0 = BOS, the next token is likely another BOS.” At inference with a single BOS, the model receives a sequence that begins identically to what it saw during pre-training but diverges from what it saw during fine-tuning—the first token after BOS is the system prompt, not another BOS. The model’s fine-tuned behavior was calibrated on the double-BOS pattern; without it, the system prompt conditioning is weaker, behavioral constraints from the system prompt are partially ignored, and instruction-following F1 drops from the base model’s 0.63 to 0.61 (fine-tuning subtracted 0.03 instead of adding 0.21).

# Diagnostic: check for double BOS in training data
# Run before training to catch this in minutes rather than weeks

from transformers import AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained("meta-llama/Meta-Llama-3-8B-Instruct")

def audit_training_sample(dataset, tokenizer, num_samples=5):
    """
    Print the first 10 decoded tokens for the first N training samples.
    A correct Llama-3 training sample begins with exactly one <|begin_of_text|>.
    A double-BOS bug produces '<|begin_of_text|><|begin_of_text|>' at the start.
    """
    print(f"BOS token id: {tokenizer.bos_token_id}")  # Should print: 128000
    print(f"BOS token string: '{tokenizer.decode([tokenizer.bos_token_id])}'")
    # Should print: '<|begin_of_text|>'

    for i, sample in enumerate(dataset[:num_samples]):
        input_ids = sample["input_ids"] if isinstance(sample, dict) else sample
        first_tokens = input_ids[:10]
        decoded = tokenizer.decode(first_tokens)
        print(f"\nSample {i}: first 10 token ids: {list(first_tokens)}")
        print(f"Decoded: '{decoded}'")

        # Check for double BOS:
        if input_ids[0] == tokenizer.bos_token_id and input_ids[1] == tokenizer.bos_token_id:
            print("⚠️  DOUBLE BOS DETECTED: tokens 0 and 1 are both BOS (128000)")
            print("   Fix: remove [tokenizer.bos_token_id] prepend from data collator")
        elif input_ids[0] == tokenizer.bos_token_id:
            print("✓  Single BOS at position 0 (correct)")
        else:
            print("⚠️  No BOS at position 0 — check add_special_tokens setting")

# Developer's configuration (broken — double BOS):
# Sample 0: first 10 token ids: [128000, 128000, 271, 2683, 374, 264, 11190, 18328, 13, 22559]
# Decoded: '<|begin_of_text|><|begin_of_text|>You are a helpful assistant. Answer'
# ⚠️  DOUBLE BOS DETECTED: tokens 0 and 1 are both BOS (128000)

# After fix (single BOS):
# Sample 0: first 10 token ids: [128000, 271, 2683, 374, 264, 11190, 18328, 13, 22559, ...]
# Decoded: '<|begin_of_text|>You are a helpful assistant. Answer concisely...'
# ✓  Single BOS at position 0 (correct)

# Training vs inference token sequence comparison:
def compare_train_inference_tokens(collator_fn, tokenizer, test_text):
    """
    Verify that training and inference produce identical token sequences.
    Any difference indicates a tokenization bug.
    """
    # Training path (through data collator)
    raw_tokenized = tokenizer.encode(test_text, add_special_tokens=True)
    train_batch = collator_fn([[raw_tokenized]])
    train_ids = train_batch["input_ids"][0].tolist()

    # Inference path (direct tokenization)
    infer_ids = tokenizer.encode(test_text, add_special_tokens=True)

    # Both should be identical
    if train_ids[:20] != infer_ids[:20]:
        print("⚠️  TRAIN/INFERENCE MISMATCH")
        print(f"Train   first 5: {train_ids[:5]}")
        print(f"Infer   first 5: {infer_ids[:5]}")
        print(f"Train decoded: '{tokenizer.decode(train_ids[:5])}'")
        print(f"Infer decoded: '{tokenizer.decode(infer_ids[:5])}'")
    else:
        print("✓  Train and inference token sequences match")

# Buggy collator output:
# Train   first 5: [128000, 128000, 271, 2683, 374]
# Infer   first 5: [128000, 271, 2683, 374, 264]
# Train decoded: '<|begin_of_text|><|begin_of_text|>You are'
# Infer decoded: '<|begin_of_text|>You are a'
# ⚠️  TRAIN/INFERENCE MISMATCH — first token differs

The fix: remove the manual BOS prepend from the data collator

The fix is to remove the [tokenizer.bos_token_id] prepend from the data collator and let tokenizer.encode(add_special_tokens=True) handle BOS insertion exclusively. After this change, training samples begin with a single [128000, content…], matching the inference pipeline exactly.

# Fixed data collator — Westfield Tokenizer Fine-Tuning Ltd.
# transformers 4.42.x, Llama-3-8B

def data_collator_fixed(batch):
    """
    Collates tokenized samples. BOS is handled by tokenizer.encode(add_special_tokens=True).
    Do NOT add an additional bos_token_id prepend here.
    """
    input_ids_list = []
    for input_ids in batch:
        # FIXED: removed [tokenizer.bos_token_id] prepend
        # tokenizer.encode(add_special_tokens=True) already prepended BOS 128000
        # input_ids[0:3] = [128000, <|system|>_token, system_text_token, ...]  ✓
        input_ids_list.append(torch.tensor(input_ids))

    padded = torch.nn.utils.rnn.pad_sequence(
        input_ids_list, batch_first=True, padding_value=tokenizer.pad_token_id or 0
    )
    labels = padded.clone()
    labels[padded == (tokenizer.pad_token_id or 0)] = -100
    return {"input_ids": padded, "labels": labels, "attention_mask": (padded != 0).long()}

# Post-fix token audit:
# Sample 0: first 10 token ids: [128000, 271, 2683, 374, 264, 11190, 18328, 13, 22559, ...]
# Decoded: '<|begin_of_text|>You are a helpful assistant. Answer concisely...'
# ✓  Single BOS at position 0 (correct)
# Training/inference token sequences match ✓

# Reference: how to handle BOS correctly for different model families
BOS_HANDLING_BY_FAMILY = {
    "llama-3": {
        "bos_token_id": 128000,
        "bos_token":    "<|begin_of_text|>",
        "approach": "use add_special_tokens=True in tokenizer.encode(); do NOT manually prepend bos_token_id",
        "note": "tiktoken-based tokenizer; add_special_tokens=True in PreTrainedTokenizerFast handles BOS",
    },
    "llama-2": {
        "bos_token_id": 1,
        "bos_token":    "",
        "approach": "use add_special_tokens=True in tokenizer.encode(); or add_special_tokens=False + manual [1] prepend (not both)",
        "note": "SentencePiece tokenizer; both approaches are valid but pick one",
    },
    "mistral": {
        "bos_token_id": 1,
        "bos_token":    "",
        "approach": "use add_special_tokens=True in tokenizer.encode(); do NOT manually prepend bos_token_id",
        "note": "SentencePiece tokenizer; same as Llama-2",
    },
    "qwen-2": {
        "bos_token_id": None,  # Qwen-2 does not use BOS in the same way
        "approach": "use tokenizer.apply_chat_template() for instruction formatting",
        "note": "chat template handles all special token insertion",
    },
}

# Recommendation: use tokenizer.apply_chat_template() for all instruction fine-tuning
# (avoids manual special token management entirely):
def prepare_sample_with_chat_template(instruction, response, tokenizer):
    """The safest approach — let the tokenizer handle all special tokens."""
    messages = [
        {"role": "system",    "content": "You are a helpful assistant. Answer concisely."},
        {"role": "user",      "content": instruction},
        {"role": "assistant", "content": response},
    ]
    text = tokenizer.apply_chat_template(
        messages,
        tokenize=False,       # get formatted string first for inspection
        add_generation_prompt=False,
    )
    print(f"Formatted (first 80 chars): {text[:80]}")
    # Verify: begins with '<|begin_of_text|>' (single BOS) for Llama-3

    tokenized = tokenizer.encode(text, add_special_tokens=False)
    # apply_chat_template already inserted special tokens in the template text
    # Use add_special_tokens=False here to avoid double-insertion
    return tokenized

# Validation log — November 2026
# transformers 4.42.x, tokenizers 0.19.x, Llama-3-8B
# 10,000 instruction-response pairs, lr=2e-5, 2 epochs, batch_size=4

# ---- Tokenization audit (before fix) ----
# Sample 0 first 5 tokens: [128000, 128000, 271, 2683, 374]
# Decoded: '<|begin_of_text|><|begin_of_text|>You are'
# → ⚠️  DOUBLE BOS DETECTED

# ---- Tokenization audit (after fix) ----
# Sample 0 first 5 tokens: [128000, 271, 2683, 374, 264]
# Decoded: '<|begin_of_text|>You are a'
# → ✓  Single BOS at position 0

# ---- Training results (original broken pipeline) ----
# Fine-tuned model, lr=2e-5, 2 epochs, 10,000 pairs:
# Training loss final:      1.08 (converged normally — training loss was not a signal)
# Base model F1 on test:    0.63
# Fine-tuned model F1:      0.61 (fine-tuning degraded the model ✗)
# Error pattern:            Model fails to follow system prompt constraints
#                           Responses are fluent but ignore instructions
# Runs tried: lr=1e-5 (F1=0.60), lr=5e-5 (F1=0.59), 20K pairs (F1=0.62),
#             3 epochs (F1=0.61) — none improved above 0.63 baseline

# ---- Training results (fixed pipeline) ----
# Fine-tuned model, lr=2e-5, 2 epochs, 10,000 pairs (re-run after collator fix):
# Training loss final:      1.06 (similar convergence)
# Fine-tuned model F1:      0.84 ← improvement confirmed ✓
# System prompt adherence:  97% (was 61% before fix)
# Response quality:         Model correctly follows all system prompt constraints

# ---- Token sequence consistency ----
# Training first 5 tokens: [128000, 271, 2683, 374, 264]
# Inference first 5 tokens: [128000, 271, 2683, 374, 264]
# → ✓  Perfect match

# ---- Re-training duration ----
# 10,000 pairs, 2 epochs, batch_size=4, A100 40GB: 1h 58m
# Total engagement: 1.5h diagnosis + 1.58h re-training wait (unbilled wait time)
# Billable: 1.5h

# ---- Hour breakdown ----
# Triage:    0.5h — fine-tuned F1 0.61 ≤ base F1 0.63; ran tokenization audit;
#            decoded first training sample: double BOS confirmed at positions 0-1;
#            training/inference comparison: tokens[0:5] differ at index 1
# Root cause: 0.25h — LLaMA-2 collator with [bos_token_id] prepend carried over;
#             add_special_tokens=True already added BOS 128000; double insertion
# Fix:       0.25h — removed [tokenizer.bos_token_id] prepend from collator;
#            audit confirmed: single BOS [128000, 271, ...]; train/infer match
# Validation: 0.5h — re-ran tokenization audit (clean); confirmed re-training would
#             fix; re-training queued (developer ran overnight); F1 0.84 confirmed
# Total:     1.5h