Advanced8 min read

Context Caching

Context caching (prompt caching) stores the model's KV state for a token-identical prefix so later calls skip re-reading tools, instructions, and stable docs — a one-byte change at the front wastes that work.

Explain like I'm new to AI

Each call, the model pre-fills: it reads every input token and builds key/value vectors (the KV cache). If 8,000 of those tokens are the same system prompt and tools as last time, recomputing them is waste.

Prompt caching keeps that prefix's KV state on the provider for a few minutes (sometimes longer). The next request with the same prefix loads it and only prefills the new suffix (the latest user message). Cached reads are much cheaper than fresh input tokens.

The catch is brutal: matching is exact prefix. Shuffle retrieval to the front, inject a timestamp, reorder tools — the cache misses.

1toolscached prefix
2system promptcached prefix
3stable docscached prefix
4history…grows, still prefix-matchable
5new questionnew suffix

Prefix is identical across calls → KV cache hit. Only the new question is prefilled.

Prompt caching is prefix matching. Put stable tools and instructions first; put anything that changes last.

Mental model

A highlighter on a textbook chapter. If the chapter is identical, you don't re-highlight. If someone inserts a sticky note on page 1, the old highlights no longer line up — you start over. The sticky note is your timestamp.

How it works

Typical render order (Anthropic-style, and the right design even on other APIs):

tools → system → messages

You mark a breakpoint (cache_control): "cache everything from the start through here." Automatic modes slide that breakpoint forward as the conversation grows. Explicit breakpoints pin a stable block (tools + system + a handbook). Providers cap how many breakpoints you get (often four).

Rules of assembly:

Stable first, variable last. Tools and instructions almost never change mid-session.
Don't put clocks, request ids, or shuffled RAG in the prefix.
Keep tool JSON key order and whitespace stable — the hash is over bytes, not "meaning."
Growing history can stay cacheable if you only append (compaction that rewrites the prefix busts it — compact carefully, then recache).

This is not the same as the in-request KV cache used while generating tokens. That cache lives inside one generation. Prompt caching reuses prefill across HTTP requests.

Real-world example

A coding agent sends the same 50 tool definitions and a 4,000-token AGENTS.md every turn, plus one new message. Without caching you pay full prefill each time. With a breakpoint after tools + system + project rules, turns 2–N mostly pay for the new tokens. Coding hosts regularly see very high cache hit rates when the prefix is disciplined.

Technical explanation

Prefill cost scales with input length. Caching stores attention K/V tensors for prefix tokens, keyed by a hash of the exact prefix. A mismatch at position N invalidates every breakpoint at N or later.

Economics: cache writes often cost a bit more than a normal input token; cache reads cost a fraction (on the order of 0.1×). Caching pays off when a prefix is read more than it is written — agents and RAG with a large static handbook are the sweet spot. A one-off call with a unique prompt should not bother.

OpenAI, Google, AWS Bedrock, and others offer analogous prefix caching; names differ (checkpoint vs breakpoint). The engineering invariant is the same: identical bytes from the left.

Common mistakes

Common mistake

Putting "today's date" or a random request id at the top of the system prompt. You silently disable caching on every call and wonder why the bill didn't drop.

Reordering tools each turn (even "to help the model").
Compacting in a way that rewrites the cached prefix every step.

When to use it

Multi-turn agents, IDEs, and RAG apps with large stable instructions or catalogs.

When NOT to use it

Single unique prompts; tiny payloads where the write surcharge isn't worth it.

Alternatives

Smaller prompts (select/compress) so prefill is cheap even without a cache; local models where you own the KV cache process.

Quick quiz

Related concepts

What is a KV Cache? — The KV cache stores past tokens' keys and values during text generation so each new token is produced without recomputing the whole sequence.
Context Compression — When the window fills, trim bulky tool output first, then compact old turns into state — prefer deletion over rewriting, and trigger on a token or turn threshold.
Token Usage, Latency, and Cost Tracking — Track input, cached, reasoning, and output tokens plus TTFT and total time — then dollars per successful task, with alerts on loops and cache busts.
Caching and Cost Optimization — Three inference caches — exact replay, prefix KV, semantic — plus $ per successful task. Key on prompt version; similar is not the same answer.
KV Cache at Serving Time — Serving is limited by KV memory, not just weights. Paged blocks and shared prefixes pack more chats; this is the engine view of the transformers KV-cache lesson.