Intermediate8 min read

Quantization for Serving

The fine-tuning quantization lesson is the idea (fewer bits). Serving quantization is three knobs — weights, activations, KV — so the model fits and decode bandwidth drops. QLoRA is still a training recipe.

Explain like I'm new to AI

Relative VRAM for the same model

~25% of 16-bit size (illustrative)

QLoRA training and GGUF/GPTQ/AWQ inference. Small quality drop; huge memory win.

Quantization stores each weight with fewer bits. Training (QLoRA) and serving (GPTQ/AWQ/GGUF) both use it — for different jobs.

Long context × concurrent users is often bigger than the weights. Quantizing KV (8-bit / FP8 / 4-bit) buys batch size. Eval: long-context recall can dip.

Serving quant is weights + activations + KV. QLoRA is a training trick. Always re-run your eval suite after you shrink bits.

Weights: GPTQ, AWQ, GGUF, bitsandbytes, FP8 checkpoints. Fit a 70B on fewer GPUs.
Activations: compute dtype (often FP8 on newer accelerators) — speed, not only disk size.
KV: often the real win for concurrency (previous lesson).

Always eval your suite after shrinking. Structured JSON, rare names, and long-context recall are the first to wobble.

Mental model

Packing for a flight: squeeze the suitcase (weights), wear lighter clothes on the plane (activations), write smaller boarding notes (KV). Different bags. QLoRA was packing while you still alter the outfit at home (train).

How it works

Calibration (AWQ/GPTQ) uses a small dataset to set scales. Garbage calibration → garbage 4-bit.

SmoothQuant-style methods protect activations. FP8 is increasingly the native serve dtype on datacenter GPUs in 2026 — sometimes you quant up from a 4-bit file into FP8 kernels. Read your engine docs; don't assume 4-bit is always fastest.

LoRA at serve time: adapters stay small; the base can be quantized. Don't mix a 4-bit base with an adapter trained at a mismatched layout without checking the engine.

Real-world example

INT4 weights fit. p95 quality on tool-call JSON drops 8 points. You keep 8-bit or FP8 for that route and 4-bit for summarization. Routing (later) + eval slices, not one global quant.

Technical explanation

Decode is memory-bound: fewer bytes per KV token ⇒ more tokens/s until you hit compute. Prefill may still be compute-bound — weight quant helps there too.

Don't confuse training QLoRA NF4 with inference AWQ. Same family of ideas, different artifacts.

Common mistakes

Common mistake

Shipping 4-bit because a blog said "lossless" without running your gold set.

Quantizing KV to 4-bit on a 128k support bot and wondering why citations drift.
Calling GGUF "the algorithm" — it's a format many local runners use.

When to use it

When weights or KV don't fit, or decode is bandwidth-starved. Start 8-bit/FP8, then 4-bit if eval holds.

When NOT to use it

Don't 4-bit a judge or a high-stakes extractor until eval says so.
Don't skip this recap and re-quantize a QLoRA training setup as if it were a server.

Alternatives

Smaller dense model, MoE with expert offload, more GPUs, shorter context.

Quick quiz

Related concepts

LoRA and QLoRA — LoRA trains two small matrices instead of the frozen weight W; QLoRA also stores W in 4-bit so a 7B+ model fine-tunes on one GPU.
KV Cache at Serving Time — Serving is limited by KV memory, not just weights. Paged blocks and shared prefixes pack more chats; this is the engine view of the transformers KV-cache lesson.

NextModel Serving and Inference Servers

Last reviewed: 2026-09-04 · Written by ByHeart AI · Reviewed by ByHeart AI