Caching and Cost Optimization
Cost optimization at inference is cache what is safe, route what is easy, and stop paying for tokens that don't finish a task. There are three caches; only one of them means "the answer is the same."
Explain like I'm new to AI
The context-engineering Context Caching lesson is prefix KV / provider prompt cache. This lesson is the cost picture plus the other two caches people smash into that word.
Hit means: Shared system+tools prefix skips prefill. Engine pages or a provider prompt-cache.
It lies when: Any edit in the prefix busts it. Don't put per-user secrets in the shared prefix (security + cache).
- Exact: same request bytes (+ auth scope) → reuse answer or KV. Safe if the world didn't change.
- Prefix KV: skip prefill of tools/instructions. Engine pages or HTTP prompt cache. Busts if the prefix moves a comma.
- Semantic: near-duplicate questions. Great for FAQs. Dangerous for "my order 441" vs "my order 442."
Cost KPI remains $ / successful task (eval + observability): cache hits, cheaper routes, fewer retries, shorter decode, less RAG over-retrieve.
Mental model
A library: the exact book, the same introduction chapter shared by many books, and a clerk who thinks you asked this last week. The clerk is semantic cache. Sometimes brilliant. Sometimes a lawsuit.
How it works
Keys: prompt_version, model, tenant, tool-catalog hash, retrieval index version. Miss those and you serve ghosts.
TTLs: FAQs days; prices minutes; user-specific never semantic-cached.
Don't cache errors or ungrounded answers. Don't cache across tenants.
Prefill-heavy apps (huge tools lists) win most from prefix cache — keep the prefix byte-stable (context engineering).
Batch/offline: sort requests that share a prefix so the engine's automatic prefix cache hits.
Real-world example
Prompt cache hit rate 0% after a "tiny" system-prompt edit every deploy. You busted the prefix. Pin versions; put volatile user text after the cache breakpoint.
Technical explanation
Provider cache discounts uncached input tokens; still log cached vs uncached (token-latency-cost). Engine prefix cache saves compute, which may or may not show as a line item.
Semantic cache is a retriever + policy. Treat hits as untrusted remembered answers — still apply ACLs.
Speculative decoding and quant reduce $ too; they're not caches.
Common mistakes
Semantic-caching all support chats by embedding similarity 0.82. You mixed two customers' orders.
- Cache key missing prompt version.
- Celebrating hit rate while quality drops (stale docs).
When to use it
- Stable prefixes: always. Exact: idempotent asks. Semantic: curated FAQ only.
When NOT to use it
- Don't semantic-cache anything with IDs, money, or "as of today."
- Don't skip RAG because "the cache had an answer last month."
Alternatives
- Smaller model + routing. Shorter outputs. Don't generate when a database row would do.
Quick quiz
Related concepts
- Context Caching — Prompt caching reuses the KV cache of a stable prefix so you don't re-prefill tools and instructions every turn — any change in that prefix busts the cache.
- Token Usage, Latency, and Cost Tracking — Track input, cached, reasoning, and output tokens plus TTFT and total time — then dollars per successful task, with alerts on loops and cache busts.
- KV Cache at Serving Time — Serving is limited by KV memory, not just weights. Paged blocks and shared prefixes pack more chats; this is the engine view of the transformers KV-cache lesson.
Last reviewed: 2026-09-04 · Written by ByHeart AI · Reviewed by ByHeart AI