Quantization
Reducing the numeric precision of a model's weights (e.g., from 16-bit to 4-bit) to cut memory and inference cost at a small accuracy trade-off.
What Is Quantization
Quantization is the process of converting a model’s weights from higher-precision floats (FP32, FP16, BF16) to lower-precision formats (INT8, INT4, even INT2). The result is a smaller, faster model that uses dramatically less GPU memory and runs faster at inference time, with a measurable but usually acceptable accuracy hit. Use it when you need to fit a model on cheaper hardware or speed up inference without retraining.
How It Works
- Post-training quantization (PTQ) — applied after training; fastest path to a quantized model. GPTQ and AWQ are the common methods for LLMs
- Quantization-aware training (QAT) — model is trained or fine-tuned with quantization simulated; preserves more accuracy but costs more
- Format families — GGUF (llama.cpp), GPTQ, AWQ, exllama, MLX (Apple), all encode quantized weights with different trade-offs
- Mixed precision — sensitive layers (attention) often kept higher precision while bulk feed-forward weights are heavily quantized
Why It Matters
A 70B model in FP16 needs ~140GB of GPU memory — multi-GPU only. The same model quantized to 4-bit fits in ~35GB and runs on a single H100 or two consumer GPUs. Quantization is the reason you can run Llama 4 or Qwen 3 locally on a Mac Studio M4 Ultra or a single RTX 4090. Almost every “local LLM” workflow depends on it.
Examples
- GGUF + llama.cpp — most common local quantization stack, used by Ollama and LM Studio
- AWQ — activation-aware quantization, popular for serving with vLLM
- GPTQ — older but still common method, supported by most inference engines
- Hugging Face bitsandbytes — the easiest path to 4-bit / 8-bit quantization in PyTorch