Name: HourTab
Author: HourTab

Question 1

What does an Axolotl developer on retainer typically do?

Accepted Answer

An Axolotl developer on monthly retainer covers YAML configuration and training setup (writing axolotl config.yaml files specifying base_model, model_type, tokenizer_type, datasets, sequence_len, sample_packing, pad_to_sequence_len, group_by_length, gradient_accumulation_steps, micro_batch_size, num_epochs, learning_rate, and LoRA parameters; understanding how axolotl preprocesses datasets into tokenized sequences and how sample_packing concatenates short examples to fill context_length for GPU efficiency; configuring flash_attention: true for long-context training; validating that packed training batches have correct per-example attention masks using axolotl's built-in batch inspection tools), attention mask debugging (diagnosing cross-example attention leakage in packed sequences — when sample_packing: true uses a simple concatenation with pad_id=0 separator and the attention mask does not block attention across example boundaries, the model can attend to tokens from the previous packed example during training; detecting this by inspecting the attention_mask tensor shape and values in a training batch and confirming that positions from one example can attend to positions in another; using flash_attention with block_diag attention or axolotl's corrected sample_packing implementation that generates per-example causal attention masks), and inference quality validation (verifying that a fine-tuned model does not exhibit context bleed — generating responses to prompts that are unambiguous in isolation and confirming the model does not reference preceding unrelated context; implementing decode_training_sample() to inspect what the model actually sees during training; measuring inference quality degradation by comparing outputs on prompts that would be affected by cross-example attention).

Question 2

What Axolotl sample_packing issues are most common on retainer?

Accepted Answer

The most common silent failure in Axolotl fine-tuning is enabling sample_packing: true without verifying that the attention mask correctly blocks cross-example attention. Sample packing is a GPU efficiency optimization that concatenates multiple short training examples into a single sequence of length context_length (e.g., 4096 tokens), separated by special tokens. This allows the GPU to process more examples per batch and improves training throughput by eliminating padding waste. In Axolotl 0.4.x, when sample_packing: true is set, axolotl uses a packing algorithm that places examples end-to-end with separator tokens. However, the attention mask in the packed sequence may be set to allow the model to attend to all previous tokens in the packed sequence — including tokens from the preceding example — rather than restricting attention to within each individual example. During training, the model can see context from the previous packed example when predicting the next token in the current example. The model learns that it can condition its output on what came before, even when those preceding tokens are from an entirely unrelated conversation. The training loss decreases normally (the model is learning, just learning partially wrong cross-example dependencies). At inference time, each query is processed as a single example with no preceding context. But the model has learned to expect context it was trained with — when given a new question, it sometimes generates text that would make sense only if there were a preceding unrelated conversation. Responses mix vocabulary, topics, or framings from the training examples that happened to be packed before the actual example. The symptom: inference outputs appear fluent but topically inconsistent, as if the model is partially answering a different question.

Question 3

What are typical Axolotl developer retainer rates?

Accepted Answer

Entry-level Axolotl developers with foundational skills (writing config.yaml files for LoRA fine-tuning on Llama-class models; configuring datasets with type: alpaca or type: sharegpt; understanding sequence_len, micro_batch_size, gradient_accumulation_steps; running axolotl preprocess and axolotl train commands on a single GPU; loading fine-tuned models with from_pretrained for evaluation; basic understanding of sample_packing as a throughput optimization) typically bill at $85 to $120 per hour. Mid-level Axolotl engineers with production experience (diagnosing sample_packing cross-example attention leakage by inspecting training batch attention masks; implementing decode_training_sample() to verify packed batch tokenization; configuring flash_attention with correct per-example causal masks; tuning LoRA parameters — r, lora_alpha, target_modules — for instruction-following quality; using group_by_length to minimize padding without cross-example attention risk; validating fine-tuned model inference quality on held-out prompts; integrating axolotl training pipelines with model serving infrastructure) typically bill at $105 to $160 per hour. Senior Axolotl architects with large-scale experience (designing multi-GPU axolotl training pipelines on A100/H100 clusters; optimizing sample_packing with correct block-diagonal attention for maximum throughput without quality degradation; capacity planning for instruction fine-tuning at scale; implementing automated per-epoch evaluation with contamination detection; managing iterative fine-tuning workflows with adapter merging and versioning) typically bill at $145 to $215 per hour. Monthly retainer ranges: $6,500 to $13,000 per month for production instruction fine-tuning reliability; $13,000 to $24,000 per month for large-scale axolotl training architecture.

Question 4

What should an Axolotl developer retainer agreement include?

Accepted Answer

An Axolotl developer retainer agreement should specify: axolotl version, base model architecture, context_length, sample_packing setting, flash_attention setting, dataset format (alpaca/sharegpt/completion), number of GPUs, LoRA configuration (r, lora_alpha, target_modules), and inference quality SLA (e.g., response coherence score on held-out evaluation set ≥ 0.85). The critical technical clause is the attention mask contract for sample_packing. If sample_packing: true is used: (1) verify that the attention mask in packed sequences blocks cross-example attention — decode a training batch and confirm that example N cannot attend to tokens from example N-1; use axolotl's built-in check_packed_sample() or implement: batch = next(iter(train_dataloader)); mask = batch['attention_mask'][0]; assert mask is causal-within-example and 0 across example boundaries; (2) prefer flash_attention: true with sample_packing: true on axolotl versions that implement correct block-diagonal attention — these versions generate per-example attention masks that prevent cross-example attention while retaining the throughput benefits; (3) if cross-example attention cannot be ruled out, fall back to sample_packing: false with group_by_length: true — this bins examples by length to minimize padding without cross-example attention risk; throughput is lower but attention semantics are correct; (4) always validate inference quality on a held-out prompt set before declaring training complete — prompts should test whether the model generates responses that would only make sense given a preceding unrelated context. The data integrity clause: every Axolotl fine-tuning pipeline should include a post-training inference validation step that runs 50 single-turn prompts and checks that outputs do not reference undefined entities that were not in the prompt — context bleed produces outputs that cite names, topics, or vocabulary not present in the evaluated prompt.

Question 5

How should Axolotl developer retainer hours be logged?

Accepted Answer

Log each Axolotl retainer session with: axolotl version, base model, context_length, sample_packing setting, flash_attention setting, dataset format, issue type (cross-example attention leakage in packed sequences — model conditions on preceding unrelated context — inference bleeds conversations; training loss does not decrease — dataset format mismatch — sharegpt vs alpaca type mismatch; adapter loading fails after axolotl training — axolotl saves adapter to merged_model path not adapter path — PeftModel cannot load; GPU OOM during training — micro_batch_size too high for context_length with sample_packing — reduce micro_batch_size or context_length; inference slower than expected — merge_model not called — LoRA overhead at inference), root cause (Northfield Instruct Fine-Tuning Ltd.; axolotl 0.4.x; Python 3.11; PyTorch 2.3.x; transformers 4.39.x; 4-GPU A100; Llama-3-8B instruction fine-tuning; 50,000 instruction examples averaging 800 tokens; context_length: 4096; sample_packing: true for GPU efficiency; axolotl 0.4.x packing concatenates examples with pad_id=0 separator; attention mask allows full causal attention within packed 4096-token sequence; model attends to tokens from preceding example during training; loss 2.4→0.71 (correct — model learned, including cross-example spurious dependencies); at inference: single example, no preceding context; model generates text referencing topics or entities from examples that would have been packed before it during training; developer configures max_steps, micro_batch_size, LoRA r and lora_alpha — 3 weeks — never inspects attention_mask tensor in a training batch to confirm per-example blocking; fix: sample_packing: false; group_by_length: true; re-run training; inference coherent on single-turn prompts; 1.5h), and hours by component (triage: 0.5h — inference outputs contain references to undefined entities; decode 10 training examples from packed batches; confirm attention mask has no cross-example blocking; root cause: 0.25h — sample_packing: true in config.yaml; axolotl 0.4.x attention mask spans full packed sequence; fix: 0.5h — set sample_packing: false; group_by_length: true; re-preprocess; retrain; validation: 0.25h — training loss converges to 0.74; single-turn inference outputs reference only prompt entities; context bleed eliminated; 1.5h total). Hours logged to HourTab with axolotl version, sample_packing setting, root cause, and fix details so the client can audit the full 1.5-hour engagement at the shared URL without logging in.

Axolotl developer on retainer: `sample_packing: true` allows cross-example attention in packed sequences — model conditions on preceding unrelated context during training — inference bleeds conversations — 3 weeks — Axolotl developer on monthly retainer

Why `sample_packing: true` causes cross-example attention leakage

The fix: `sample_packing: false` with `group_by_length: true`

Validation

Related retainer posts

Axolotl developer on retainer: sample_packing: true allows cross-example attention in packed sequences — model conditions on preceding unrelated context during training — inference bleeds conversations — 3 weeks — Axolotl developer on monthly retainer

Why sample_packing: true causes cross-example attention leakage

The fix: sample_packing: false with group_by_length: true

Validation

Related retainer posts

Axolotl developer on retainer: `sample_packing: true` allows cross-example attention in packed sequences — model conditions on preceding unrelated context during training — inference bleeds conversations — 3 weeks — Axolotl developer on monthly retainer

Why `sample_packing: true` causes cross-example attention leakage

The fix: `sample_packing: false` with `group_by_length: true`