What is Inference?
Inference is running a finished model to get an answer: first prefill (eat the prompt), then decode (emit tokens). Training changed the weights; inference uses them.
Explain like I'm new to AI
The foundations lesson Training vs Inference is the fork: learn vs use. This category is use at scale — GPUs, queues, money, milliseconds.
The prompt is ingested. This is compute-heavy and sets time-to-first-token.
The whole prompt in parallel. K and V for every prompt token are written into the cache.
A chat reply is not one magic step:
- Queue — wait for a slot (often the silent killer of TTFT).
- Prefill — the whole prompt runs in parallel. Heavy compute. Ends when the first token can be sampled. That's most of time to first token.
- Decode — one token, then another. The KV cache (transformers lesson) makes this not quadratic. The GPU is often memory-bound, rereading cache. Streaming is this loop.
2026 reasoning models spend a lot of decode on hidden tokens you never show. Same physics, bigger bill (observability token lesson).
Mental model
A restaurant: training is writing the cookbook (rare). Inference is cooking tonight. Prefill is mise en place for this ticket; decode is plating bite by bite.
How it works
Weights stay frozen (unless you load a LoRA). Input tokens → hidden states → next-token distribution → sample (temperature lesson) → append → repeat until stop.
Throughput is how many tokens (or successful requests) the fleet emits. Latency is how long this user waited. They move in opposite directions when you batch (next lessons).
Self-hosting vs an API: same phases. The API hides the GPU; you still pay for prefill, decode, and retries.
Real-world example
A support bot "feels slow." Traces show prefill is 200ms but queue is 1.8s. Buying a smarter model would not help. More replicas or better batching would.
Technical explanation
Prefill scales with prompt length (attention over the prompt). Decode scales with output length × cache size. Long context hurts both: more prefill, fatter KV per token of decode.
Speculative decoding, quantization, routing, and prefix cache are all tricks on these two phases — coming up.
Security still applies: inference is where injection becomes a tool call. Caps on tokens are also denial-of-wallet controls.
Common mistakes
Calling every GPU-second "training." If the weights aren't updating, it's inference.
- Optimizing decode when the user is waiting in queue.
- Ignoring reasoning-token decode when comparing "same" models.
When to use it
- Every production answer, embedding call, and reranker is inference. Design for prefill vs decode explicitly.
When NOT to use it
- Don't run huge interactive decode on a box you sized for training jobs (different utilization pattern).
- Don't "infer" to update facts — that's RAG or tools.
Alternatives
- Distill / smaller model / retrieve-then-extract without a giant decode. Still inference, just cheaper.
Quick quiz
Related concepts
- What is an LLM? — An LLM is a large neural network trained on huge amounts of text to predict and generate language.
- What is a KV Cache? — The KV cache stores past tokens' keys and values during text generation so each new token is produced without recomputing the whole sequence.
- Quality, Latency, Cost, and Reliability — Production AI is a tradeoff of quality, latency, cost, and reliability. Optimize dollars per successful task under an SLO — not always the smartest model.
Last reviewed: 2026-09-04 · Written by ByHeart AI · Reviewed by ByHeart AI