What is a KV Cache?
A KV cache stores the keys and values of past tokens so that, while generating text, the model only computes the newest token instead of redoing all the previous work.
Explain like I'm new to AI
LLMs generate text one token at a time, and each new token needs to attend back over all the tokens before it. Done naively, that means re-processing the entire sentence every single step — hugely wasteful and slow.
The KV cache is the fix. During attention, every past token produces a key and a value. Since those don't change once computed, we store (cache) them and reuse them for every future token. Only the brand-new token has to be computed.
Step through it — watch the savings grow:
Generate tokens one at a time. Cached tokens are reused; only the newest token is computed.
Mental model
Imagine writing a long story and, before adding each new sentence, re-reading the entire story from the start. Exhausting. Instead, you keep a running set of notes (the cache) about everything so far, so you only think about the next sentence. That's the KV cache.
How it works
- As the prompt is processed, each token's key (K) and value (V) vectors are computed and saved in the cache.
- To generate the next token, the model computes a query for just that one new token.
- It attends over the cached K and V (all past tokens) — no recomputation.
- The new token's own K and V are appended to the cache, ready for the following step.
Without the cache, generating n tokens costs work proportional to 1 + 2 + … + n (≈ n²). With the cache, it's proportional to n. That's the difference between a chatbot that feels instant and one that crawls.
Real-world example
When a chatbot streams a long answer smoothly, the KV cache is why each word appears quickly and at a steady pace — the model isn't re-reading its whole reply for every new word.
Technical explanation
The KV cache trades memory for speed: its size grows with batch × layers × heads × head_dim × sequence_length, so long conversations can consume large amounts of GPU memory. This is a major reason large context windows are costly to serve. Mitigations include:
- GQA / MQA: share keys/values across heads to shrink the cache.
- PagedAttention (vLLM): manage cache memory in pages to reduce waste and enable more concurrent users.
- Quantized / windowed caches: store the cache in lower precision or keep only recent tokens.
Note the cache only helps inference/generation — it isn't used the same way during training, where the full sequence is processed at once.
Common mistakes
Confusing the KV cache with the context window. The context window is how many tokens the model can consider; the KV cache is the stored keys/values that make generating within that window fast.
- Assuming the cache is free — it consumes significant memory that scales with sequence length.
- Thinking it speeds up training — it's an inference-time optimization.
When to use it
- Always, for autoregressive text generation — it's standard in every serving stack.
When NOT to use it
- Non-generative passes (e.g. computing a single embedding), where there's nothing to cache across steps.
Alternatives
- The optimization itself is near-universal; the alternatives (GQA, MQA, paged/quantized caches) make it cheaper, not optional.
Quick quiz
Related concepts
- What is Self-Attention? — Self-attention is attention applied within a single sequence, letting every word gather context from every other word in the same text.
- What is Multi-Head Attention? — Multi-head attention runs several attention computations in parallel, each learning to focus on a different kind of relationship.
- Context Caching — Prompt caching reuses the KV cache of a stable prefix so you don't re-prefill tools and instructions every turn — any change in that prefix busts the cache.
- KV Cache at Serving Time — Serving is limited by KV memory, not just weights. Paged blocks and shared prefixes pack more chats; this is the engine view of the transformers KV-cache lesson.
Last reviewed: 2026-08-30 · Written by ByHeart AI · Reviewed by ByHeart AI