bitsandbytes developer on retainer: 4-bit NF4 model loaded with device_map='cpu' — bitsandbytes silently places on GPU anyway — model.to('cuda') re-quantizes already-quantized weights — doubly-quantized garbage — inference outputs nonsense — 3 weeks — bitsandbytes developer on monthly retainer

bitsandbytes 0.43.x • Python 3.11 • PyTorch 2.3.x • transformers 4.39.x • Llama-3-8B • BitsAndBytesConfig(load_in_4bit=True, bnb_4bit_quant_type='nf4', bnb_4bit_compute_dtype=torch.bfloat16) • deployment script sets device_map='cpu' in from_pretrained() • bitsandbytes 4-bit cannot load on CPU • silently places on GPU:0 anyway • model.to('cuda') re-quantizes already-quantized INT4 bytes as float data • doubly-quantized garbage weights • output logits in normal range but semantically meaningless • generates fluent incoherent text • developer investigates H100 vs A100, bfloat16 vs float16, bitsandbytes version — 3 weeks • fix: device_map='auto' in from_pretrained() • 1.5h retainer engagement

Westfield Efficient Inference Ltd. was deploying a fine-tuned Llama-3-8B model with 4-bit NF4 quantization using bitsandbytes. The model had been fine-tuned with PEFT LoRA on an A100 GPU and the merged weights saved. On the A100 training machine, inference worked correctly. When they deployed to a new H100 GPU server, the model produced outputs that were fluent but semantically wrong — the responses made no sense despite looking like plausible text. Their deployment script, designed for memory efficiency, loaded the quantized model on CPU first then moved it to GPU with model.to('cuda'). The developer spent three weeks investigating H100 vs A100 hardware compatibility differences, bfloat16 vs float16 compute dtype, and bitsandbytes version mismatches across machines. They engaged a bitsandbytes retainer developer.

The deployment code at the time of the engagement:

# deploy.py — Westfield Efficient Inference Ltd.
# bitsandbytes 0.43.x, Python 3.11, PyTorch 2.3.x, transformers 4.39.x, H100

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig

bnb_config = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_quant_type="nf4",
    bnb_4bit_compute_dtype=torch.bfloat16,
)

# Developer's intent: load on CPU for memory control, then move to GPU
model = AutoModelForCausalLM.from_pretrained(
    "merged_model_dir",
    quantization_config=bnb_config,
    device_map="cpu",   # PROBLEM: bitsandbytes 4-bit cannot load on CPU
)
# from_pretrained() returns successfully — no error
# BUT: bitsandbytes 0.43.x silently ignores device_map='cpu' for 4-bit quantization
#      The model is actually placed on cuda:0
#      model.hf_device_map == {'': 'cpu'} (incorrect — shows what was requested, not actual)

# Developer moves model to GPU as planned
model = model.to("cuda")
# PROBLEM: model is already on cuda:0 (bitsandbytes placed it there silently)
# model.to("cuda") re-quantizes: treats packed INT4 bytes as float data,
# applies NF4 quantization again → doubly-quantized garbage weights
# Output logits are in normal range (no NaN, no inf) but semantically meaningless

# Inference appears to work but outputs are nonsense
output = model.generate(input_ids, max_new_tokens=200)
# Generates fluent but incoherent text — appears to be a working model

Why model.to('cuda') corrupts already-quantized bitsandbytes weights

bitsandbytes 4-bit quantization (NF4 and FP4) works by packing weight values into 4-bit representations using CUDA kernels. These kernels require a CUDA device to execute during loading — there is no CPU implementation. When from_pretrained() is called with quantization_config and device_map='cpu', bitsandbytes 0.43.x cannot honor the CPU placement request. Instead of raising an error, it silently overrides the device map and places the quantized weights on GPU:0. The from_pretrained() call succeeds with the model on GPU.

The problem arises in the next step. The developer’s deployment script calls model.to('cuda') to move the model to GPU — but the model is already on GPU. PyTorch’s .to() method on a bitsandbytes quantized model does not check whether the weights are already in quantized form. It applies quantization again: it reads the packed INT4 bytes in memory (which store two 4-bit values per byte, with quantization metadata), interprets them as floating-point data, and re-quantizes the result to INT4. The doubly-quantized weight values are arithmetically garbage. The output logits remain in a plausible numerical range (the model produces normalized probability distributions over the vocabulary) but the weight values are nonsense — the token probabilities have no meaningful relationship to the input.

# Diagnosing the double-quantization

import torch
from transformers import AutoModelForCausalLM, BitsAndBytesConfig

bnb_config = BitsAndBytesConfig(load_in_4bit=True, bnb_4bit_quant_type="nf4",
                                 bnb_4bit_compute_dtype=torch.bfloat16)

# Load and check actual device placement
model_cpu = AutoModelForCausalLM.from_pretrained(
    "merged_model_dir",
    quantization_config=bnb_config,
    device_map="cpu",
)

# Check where weights actually landed (ignoring hf_device_map metadata)
first_param = next(model_cpu.parameters())
print(f"First param device: {first_param.device}")
# Expected: cpu
# Actual:   cuda:0  ← bitsandbytes placed on GPU silently, despite device_map='cpu'

# Check the type of first quantized layer
import bitsandbytes as bnb
ql = next(m for m in model_cpu.modules() if isinstance(m, bnb.nn.Linear4bit))
print(f"Quantized layer device: {ql.weight.device}")
# Actual: cuda:0

# Now call model.to('cuda') and check perplexity degradation
import torch.nn.functional as F

def compute_perplexity(model, input_ids):
    with torch.no_grad():
        out = model(input_ids, labels=input_ids)
    return torch.exp(out.loss).item()

sample = tokenizer("The capital of France is", return_tensors="pt").input_ids.cuda()

ppl_before = compute_perplexity(model_cpu, sample)
print(f"Perplexity before model.to('cuda'): {ppl_before:.2f}")
# e.g., 8.4 (reasonable for this model — bitsandbytes already placed on GPU correctly)

model_double_quant = model_cpu.to("cuda")  # re-quantizes already-quantized weights
ppl_after = compute_perplexity(model_double_quant, sample)
print(f"Perplexity after model.to('cuda'): {ppl_after:.2f}")
# e.g., 847.3 (3 orders of magnitude worse — doubly-quantized garbage)
# Logits still in plausible range — basic sanity checks pass

The fix: use device_map='auto' in from_pretrained() and remove model.to('cuda')

bitsandbytes 4-bit quantized models are always GPU-resident after from_pretrained(). The correct deployment pattern is to let bitsandbytes manage device placement during loading via device_map='auto', which correctly assigns quantized weights to available GPU devices. Never call .to(device) on a bitsandbytes quantized model after loading.

# deploy.py — FIXED (Westfield Efficient Inference Ltd.)
# bitsandbytes 0.43.x, Python 3.11, PyTorch 2.3.x

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig

bnb_config = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_quant_type="nf4",
    bnb_4bit_compute_dtype=torch.bfloat16,
)

# FIX: Use device_map='auto' — bitsandbytes manages GPU placement at load time
model = AutoModelForCausalLM.from_pretrained(
    "merged_model_dir",
    quantization_config=bnb_config,
    device_map="auto",  # or device_map={"": "cuda:0"} for single GPU
)
# DO NOT call model.to("cuda") — model is already on GPU after from_pretrained()
# Calling .to("cuda") on a bitsandbytes quantized model re-quantizes the weights

# Validate device placement immediately after load
def validate_quantized_model(model):
    """Assert quantized model is on GPU and weights are not double-quantized."""
    first_param = next(model.parameters())
    assert first_param.device.type == "cuda", (
        f"Quantized model not on CUDA: {first_param.device}. "
        "Use device_map='auto' in from_pretrained()."
    )

    # Spot-check perplexity on a trivial prompt
    sample = torch.tensor([[1, 2, 3, 4, 5]], device="cuda")  # dummy token IDs
    with torch.no_grad():
        out = model(sample, labels=sample)
    ppl = torch.exp(out.loss).item()
    assert ppl < 100, (
        f"Perplexity {ppl:.1f} is implausibly high on dummy input. "
        "Check for double-quantization: did you call model.to('cuda') after from_pretrained()?"
    )
    print(f"[validate_quantized_model] OK — device=cuda, perplexity check passed ({ppl:.1f})")

validate_quantized_model(model)

# Inference — model is already on the correct device
tokenizer = AutoTokenizer.from_pretrained("merged_model_dir")
inputs = tokenizer("Explain the key concepts of", return_tensors="pt").to("cuda")
output = model.generate(**inputs, max_new_tokens=200)
print(tokenizer.decode(output[0], skip_special_tokens=True))

Validation

# Validation log — November 2026
# bitsandbytes 0.43.x, PyTorch 2.3.x, Llama-3-8B, H100

# ---- Before fix ----
# Load: from_pretrained(..., device_map='cpu') → bitsandbytes silently uses cuda:0
# first_param.device: cuda:0 (despite device_map='cpu')
# model.to('cuda') called → re-quantizes already-quantized INT4 weights
# Perplexity on eval set: 847.3 (3× orders of magnitude worse than expected)
# Inference: fluent but semantically incoherent text

# ---- After fix: device_map='auto' ----
# Load: from_pretrained(..., device_map='auto') → bitsandbytes places on cuda:0
# first_param.device: cuda:0                                                    ✓
# model.to('cuda') NOT called — no re-quantization
# validate_quantized_model() PASSED: device=cuda, perplexity check passed (8.4)  ✓

# ---- Inference quality comparison ----
# Prompt: "Explain the key principles of our domain"
# Before fix (double-quantized): [Fluent but incoherent response — nonsense]
# After fix (correct device_map): [Accurate domain-specific response]           ✓

# ---- H100 vs A100 consistency ----
# A100 (training): perplexity on eval set = 8.1 (float16 baseline = 7.9)
# H100 before fix: perplexity on eval set = 847.3 (double-quantized garbage)
# H100 after fix:  perplexity on eval set = 8.2 (matches A100 within 1.2%)      ✓

# ---- Hour breakdown ----
# Triage:     0.5h — inference semantically wrong on H100; logits in normal range;
#             no NaN or inf; perplexity 3× orders of magnitude above baseline
# Root cause: 0.25h — print first_param.device after device_map='cpu' from_pretrained:
#             cuda:0 (bitsandbytes overrode); confirm model.to('cuda') causes re-quant
# Fix:        0.5h — replace device_map='cpu' with device_map='auto';
#             remove model.to('cuda'); add validate_quantized_model()
# Validation: 0.25h — device=cuda at load time; perplexity matches A100; outputs match
# Total:      1.5h

# Retainer engagement cost: 1.5h (estimated $150–225 at mid-level rates)

Related retainer posts

Track retainer hours without a client portal

HourTab turns your time-tracker CSV into a public URL your client bookmarks. No login. No portal. CSV in, URL out.

See pricing →