What is Quantization?
Quantization is shrinking each weight from 16-bit-ish numbers to 8- or 4-bit codes so the model uses less memory and often runs faster — with a little rounding error.
Explain like I'm new to AI
Weights are numbers. 16-bit (bf16/fp16) is the usual training precision. Quantization packs those numbers into fewer bits, like compressing a photo: smaller file, slightly less detail.
~25% of 16-bit size (illustrative)
QLoRA training and GGUF/GPTQ/AWQ inference. Small quality drop; huge memory win.
Two different jobs people mix up:
- Serve / inference: GPTQ, AWQ, GGUF, bitsandbytes — run a 70B on a smaller GPU or a laptop.
- Train: QLoRA keeps the frozen base 4-bit while adapters learn. That's quantization in the service of fine-tuning.
Same idea (fewer bits), different moment in the lifecycle.
Mental model
A pantry of spices in huge glass jars (16-bit) vs the same spices in small tins (4-bit). You still cook; a pinch might be a hair less precise. For most dishes you won't notice. For a soufflé (fragile eval), you might keep the jars.
How it works
A scale maps a small integer back to an approximate real. Calibration (GPTQ/AWQ) uses a bit of data to choose scales that hurt loss less. GGUF is a file format popular for local runners. KV cache can be quantized too at generation time (separate from weight quant).
Quality drop is task-dependent. Always eval your suite after quantizing — especially math, rare names, and structured output.
Real-world example
A 8B instruct model in 4-bit GGUF on a laptop for a demo. Production might serve AWQ 4-bit on one GPU instead of two. Fine-tuning still happens with QLoRA, then you export a quantized artifact for serving.
Technical explanation
Memory for weights ≈ params × bytes_per_weight. 4-bit is ~4× smaller than 16-bit (plus overhead). Activations and KV cache still need RAM — so "4-bit 70B" is not 4× smaller end-to-end.
Don't train LoRA adapters in 4-bit. Don't assume QLoRA training artifacts are automatically the best GGUF; convert deliberately.
Activation-aware methods (AWQ) protect outlier weights that carry more signal.
Common mistakes
Quantizing, skipping eval, shipping. A 2% MMLU drop might be a 20% hit on your JSON schema or legal clause.
- Mixing up QLoRA (train) with GGUF (serve).
- 2-bit because "smaller is better" — quality falls off a cliff.
When to use it
- Fit a model on available GPUs; local apps; cheaper tokens at scale after you measure quality.
When NOT to use it
- You already fit in 16-bit and eval is razor-thin. Don't 4-bit a model as a substitute for a smaller architecture if quality dies.
Alternatives
- Distillation to a smaller dense model; speculative decoding for speed without 4-bit; more GPUs.
Quick quiz
Related concepts
- LoRA and QLoRA — LoRA trains two small matrices instead of the frozen weight W; QLoRA also stores W in 4-bit so a 7B+ model fine-tunes on one GPU.
- What is a KV Cache? — The KV cache stores past tokens' keys and values during text generation so each new token is produced without recomputing the whole sequence.
- Quantization for Serving — Serving quantization shrinks weights, activations, and KV so more model and more users fit — it is not QLoRA. Re-run eval after you drop bits.
Last reviewed: 2026-09-04 · Written by ByHeart AI · Reviewed by ByHeart AI