Inference
Serving models efficiently: latency, throughput, and cost.
- What is Inference?Beginner
Inference is using a trained model to produce outputs — prefill the prompt, then decode tokens. Chat latency is those two phases plus queue, not "the GPU being on."
- Batching and Continuous BatchingAdvanced
Static batches wait for the slowest request. Continuous batching fills GPU slots at every token step; chunked prefill keeps long prompts from stalling streams.
- KV Cache at Serving TimeAdvanced
Serving is limited by KV memory, not just weights. Paged blocks and shared prefixes pack more chats; this is the engine view of the transformers KV-cache lesson.
- Quantization for ServingIntermediate
Serving quantization shrinks weights, activations, and KV so more model and more users fit — it is not QLoRA. Re-run eval after you drop bits.
- Model Serving and Inference ServersIntermediate
An inference server is API plus scheduler plus engine — OpenAI-shaped HTTP is the lingua franca, not the architecture. Scale on queue and KV memory, not CPU.
- Latency vs ThroughputIntermediate
Latency is one user's wait (TTFT, ITL, queue); throughput is tokens per second per GPU. Bigger batches raise throughput and hurt latency — pick the SLO first.
- Model Routing and FallbacksIntermediate
Send easy turns to a cheap model, hard turns to a strong one, and failures to a fallback — route with eval slices, not hallway demos, and don't hedge every call.
- Caching and Cost OptimizationIntermediate
Three inference caches — exact replay, prefix KV, semantic — plus $ per successful task. Key on prompt version; similar is not the same answer.
- GPU Memory and Distributed InferenceAdvanced
When weights plus KV exceed one GPU, you split tensors, pipeline layers, or — in 2026 — disaggregate prefill from decode. The interconnect becomes the bottleneck.