AutoAWQ developer on retainer: Llama-3-8B quantized with version="GEMM" — vLLM 0.3.x AWQ serving requires GEMV kernel — vLLM loads unquantized float16 silently — GPU memory 14 GB not 5 GB — 3 weeks — AutoAWQ developer on monthly retainer

autoawq 0.2.x • Python 3.11 • NVIDIA RTX 3090 24 GB • Llama-3-8B INT4 W4A16 • AwqConfig(version="GEMM") • vllm.LLM(quantization="awq") • vLLM loads unquantized float16 silently • GPU memory 14 GB not 5 GB • developer checks vllm version, autoawq version, available GPU memory, quantization flag — 3 weeks • fix: re-quantize with version="GEMV" • GPU memory 14 GB→5 GB • 1.5h retainer engagement

Southfield AutoAWQ Quant Ltd. built a production LLM serving stack for Llama-3-8B INT4 W4A16 using AutoAWQ 0.2.x for quantization and vLLM 0.3.x for serving. The developer quantized the model locally, verified inference with the AutoAWQ Python API, confirmed correct outputs, and deployed the quantized model to a vLLM serving endpoint. After deployment, GPU memory consumption on the NVIDIA RTX 3090 remained at 14 GB—the same as an unquantized float16 model—rather than the expected ~5 GB for a W4A16 INT4 quantized model. Inference outputs were correct. No errors appeared in vLLM logs. The developer spent three weeks investigating: checking vllm and autoawq package versions (both matched production requirements), verifying the quantization="awq" flag was set, checking available GPU memory, and re-running the quantization pipeline. GPU memory never moved from 14 GB. The developer engaged an AutoAWQ retainer developer after three weeks of configuration audits had not reduced memory consumption.

The quantization and serving pipeline at the time of the engagement:

# quantize.py — Southfield AutoAWQ Quant Ltd.
# autoawq 0.2.x, Python 3.11, NVIDIA RTX 3090 24GB
# Llama-3-8B INT4 W4A16

from awq import AutoAWQForCausalLM
from transformers import AutoTokenizer

model_path = "meta-llama/Meta-Llama-3-8B"
quant_path = "./llama3-8b-awq-gemm"

tokenizer = AutoTokenizer.from_pretrained(model_path, trust_remote_code=True)
model = AutoAWQForCausalLM.from_pretrained(
    model_path, low_cpu_mem_usage=True, use_cache=False
)

# BUG: version="GEMM" produces GEMM-format weight packing
# vLLM 0.3.x AWQ integration expects GEMV-format weight packing
# These two formats use different weight tensor layouts
quant_config = {
    "zero_point": True,
    "q_group_size": 128,
    "w_bit": 4,
    "version": "GEMM",   # ← correct for standalone AutoAWQ inference
                          # ← INCORRECT for vLLM serving (requires GEMV)
}
model.quantize(tokenizer, quant_config=quant_config)
model.save_quantized(quant_path)
tokenizer.save_pretrained(quant_path)

# Verification with AutoAWQ direct inference — works correctly:
model_check = AutoAWQForCausalLM.from_quantized(
    quant_path, fuse_layers=True
)
# ↑ AutoAWQ's own inference uses GEMM-format kernels — no issue here
# ↑ Outputs are correct; developer validates against float16 baseline ✓

# --- serve.py ---
from vllm import LLM, SamplingParams

llm = vllm.LLM(
    model=quant_path,
    quantization="awq",   # ← developer expects AWQ INT4 kernels
    dtype="float16",
    gpu_memory_utilization=0.90,
)

# vLLM 0.3.x loads the quantize_config.json and reads version="GEMM"
# vLLM's AWQ integration is built for GEMV weight tensor layout
# GEMM layout is incompatible: weight tensors have different packing dimensions
# vLLM silently falls back to loading float16 weights instead of INT4 weights
# No exception is raised; a debug-level log line is emitted (suppressed by default)
#
# After model load:
# torch.cuda.memory_allocated() = 14.2 GB  ← float16 Llama-3-8B
# Expected AWQ INT4 W4A16:          ~5.0 GB ← not achieved
#
# Inference: correct (float16 produces correct outputs)
# Throughput: same as unquantized float16 (no INT4 kernel benefit)

Why version="GEMM" and version="GEMV" are incompatible with vLLM

AutoAWQ 0.2.x supports two quantization kernel formats selected via the version field in quant_config:

GEMM targets high-throughput batch inference using the GEMM (General Matrix Multiply) CUDA kernel. It packs 4-bit weights into a tensor layout optimized for the GEMM kernel's access pattern: weights are packed as 8 INT4 values per INT32, arranged for coalesced memory access during matrix multiplication over large batch dimensions. This format is used by AutoAWQ's own inference path (from_quantized()) and achieves best performance for batch sizes ≥4.

GEMV targets autoregressive decoding (token-by-token generation) where the effective batch dimension per decode step is 1. It uses the GEMV (General Matrix-Vector Multiply) kernel, which packs weights in a different layout optimized for single-row matrix-vector products. This is the format vLLM's AWQ integration expects because vLLM's PagedAttention manages batching internally using its own CUDA kernels, not through matrix dimension batching.

When a model is saved with version="GEMM", the weight tensors in the saved .pt files have GEMM packing layout. When vLLM 0.3.x loads a model with quantization="awq", it reads the quantize_config.json, checks the version field, and attempts to load GEMV-format weights. Finding GEMM-layout tensors, vLLM cannot apply its AWQ INT4 kernels. Rather than raising a shape mismatch exception (the packed INT32 tensors have the same outer dimensions in both formats), vLLM 0.3.x silently skips the INT4 kernel path and loads the model in float16, dequantizing the weights to full precision during load. The result is a correctly-functioning model that uses float16 memory footprint and float16 inference speed.

# Demonstrating the memory footprint difference between GEMM and GEMV quantization

import torch
from vllm import LLM

# Load GEMM-quantized model (developer's original — silent float16 fallback):
llm_gemm = LLM(model="./llama3-8b-awq-gemm", quantization="awq")
mem_gemm = torch.cuda.memory_allocated() / 1e9
print(f"GEMM-quantized (vLLM): {mem_gemm:.1f} GB")
# Output: GEMM-quantized (vLLM): 14.2 GB  ← float16 fallback: no INT4 benefit

del llm_gemm
torch.cuda.empty_cache()

# Load GEMV-quantized model (fix — AWQ INT4 kernels active):
llm_gemv = LLM(model="./llama3-8b-awq-gemv", quantization="awq")
mem_gemv = torch.cuda.memory_allocated() / 1e9
print(f"GEMV-quantized (vLLM): {mem_gemv:.1f} GB")
# Output: GEMV-quantized (vLLM): 5.1 GB   ← AWQ INT4 W4A16: correct

# How to detect the silent fallback without re-quantizing:
# Check quantize_config.json in the saved model directory:
import json
with open("./llama3-8b-awq-gemm/quantize_config.json") as f:
    config = json.load(f)
print(f"Quantization version: {config.get('version')}")
# Output: Quantization version: GEMM  ← must be GEMV for vLLM serving

# Startup assertion to catch silent fallback in production:
def assert_awq_active(model_path: str, expected_max_gb: float = 8.0):
    """Raise if vLLM loaded the AWQ model in float16 fallback mode."""
    llm = LLM(model=model_path, quantization="awq")
    mem_gb = torch.cuda.memory_allocated() / 1e9
    assert mem_gb < expected_max_gb, (
        f"vLLM loaded {mem_gb:.1f}GB — expected <{expected_max_gb}GB for INT4 AWQ. "
        f"Check quantize_config.json: version must be 'GEMV', not 'GEMM'."
    )
    return llm

The fix: re-quantize with version="GEMV"

The fix requires re-running the quantization pipeline with version="GEMV" in the quant config. No other changes are needed to the vLLM serving configuration or model architecture.

# quantize.py — FIXED (Southfield AutoAWQ Quant Ltd.)
# autoawq 0.2.x, Python 3.11, NVIDIA RTX 3090 24GB
# Llama-3-8B INT4 W4A16

from awq import AutoAWQForCausalLM
from transformers import AutoTokenizer

model_path = "meta-llama/Meta-Llama-3-8B"
quant_path = "./llama3-8b-awq-gemv"   # ← new output path

tokenizer = AutoTokenizer.from_pretrained(model_path, trust_remote_code=True)
model = AutoAWQForCausalLM.from_pretrained(
    model_path, low_cpu_mem_usage=True, use_cache=False
)

# FIXED: version="GEMV" produces GEMV-format weight packing
# Compatible with vLLM's AWQ integration
quant_config = {
    "zero_point": True,
    "q_group_size": 128,
    "w_bit": 4,
    "version": "GEMV",   # ← required for vLLM serving
}
model.quantize(tokenizer, quant_config=quant_config)
model.save_quantized(quant_path)
tokenizer.save_pretrained(quant_path)

# --- serve.py (unchanged) ---
from vllm import LLM, SamplingParams

llm = LLM(
    model="./llama3-8b-awq-gemv",   # ← point to GEMV-quantized model
    quantization="awq",
    dtype="float16",
    gpu_memory_utilization=0.90,
)

# After fix:
# vLLM reads quantize_config.json: version="GEMV" ✓
# vLLM loads GEMV-format weight tensors with AWQ INT4 kernels
# torch.cuda.memory_allocated() = 5.1 GB  ← W4A16 INT4 ✓
# Inference: correct at 2.6× higher throughput vs float16 (fewer memory reads)

# Which version to use — decision guide:
#   version="GEMV"  → use when serving with vLLM, TGI AWQ integration, or any
#                      framework that implements its own CUDA batching kernels
#   version="GEMM"  → use when serving with AutoAWQ's own from_quantized()
#                      inference path, especially for batch_size >= 4
#
# When in doubt: use version="GEMV" — it works correctly with both AutoAWQ
# direct inference (slower for large batches) and vLLM serving

Validation

# Validation log — November 2026
# autoawq 0.2.x, Llama-3-8B INT4 W4A16, vLLM 0.3.x
# NVIDIA RTX 3090 24GB

# ---- Before fix (GEMM-quantized model in vLLM) ----
# GPU memory post-load:  14.2 GB   ← float16 fallback
# Expected (INT4 W4A16): ~5.0 GB   ← not achieved
# Inference outputs:     correct    ← float16 inference is correct
# Throughput:            4,200 tok/s (float16 — no AWQ kernel benefit)
# Output quality:        PPL 7.8 (float16 baseline)

# ---- After fix (GEMV-quantized model in vLLM) ----
# GPU memory post-load:  5.1 GB    ← AWQ INT4 W4A16 ✓
# Throughput:            11,100 tok/s (AWQ INT4 kernels active) ✓
# Output quality:        PPL 8.0  (AWQ quantization adds minor accuracy cost) ✓

# Accuracy regression test (10,000-sample eval set):
#   float16 baseline:          PPL 7.8,  MMLU 65.2%
#   GEMV AWQ INT4 (fix):       PPL 8.0,  MMLU 64.8%  ← acceptable regression ✓
#   GEMM AWQ INT4 (AutoAWQ):   PPL 8.0,  MMLU 64.8%  ← same quantization quality ✓
# (GEMM and GEMV quantization use the same calibration algorithm;
#  only the kernel weight packing differs — accuracy is identical)

# ---- Hour breakdown ----
# Triage:     0.5h — GPU memory 14.2GB post-load; quantization='awq' set;
#             no vLLM errors; quantize_config.json inspected; version='GEMM' found;
#             vLLM AWQ docs: GEMV required; float16 fallback mechanism confirmed
# Root cause: 0.25h — GEMM vs GEMV weight packing format incompatibility;
#             vLLM 0.3.x silently falls back to float16 on GEMM-format model
# Fix:        0.5h — re-quantize with version='GEMV'; save to new path; reload in vLLM
# Validation: 0.25h — GPU memory 14.2GB→5.1GB; throughput 4,200→11,100 tok/s;
#             PPL regression 7.8→8.0 (acceptable); outputs correct
# Total:      1.5h

# Retainer engagement cost: 1.5h (estimated $128–203 at mid-level rates)

Related retainer posts

Track retainer hours without a client portal

HourTab turns your time-tracker CSV into a public URL your client bookmarks. No login. No portal. CSV in, URL out.

See pricing →