Model Routing and Fallbacks
Routing picks which model (or size, or reasoning profile) handles this turn. Fallbacks pick what happens when that call dies. The default "always the smartest model" is how you miss SLOs and invoices.
Explain like I'm new to AI
What kind of turn is this?
Route: Small / cheap model (or cache). Escalating to a frontier model wastes $ and TTFT.
Patterns that work in 2026:
- Cascade: try small/fast; escalate if the small model is unsure, needs tools, or fails a cheap validator (JSON schema).
- Task route: FAQ vs coding vs long reasoning — known up front from the product surface, not from a mystical classifier you never eval.
- Fallback: timeout, 429, empty, provider incident → retry with jitter, then another model, then a degraded honest message.
- Shadow: new model scores on live traffic without user visibility (eval + observability).
Hedging (send two models, take the first) is a latency trick that doubles cost. Use for a tiny VIP slice, not default.
Mental model
A hospital triage nurse: band-aids don't go to surgery; chest pain doesn't wait for the intern. The nurse is policy + a few measurements, not "whoever looks smartest."
How it works
- Write slices on the gold set (easy/hard/tools). A router that wins overall but fails the hard slice is a fail.
- Features for a learned router: length, tool-need heuristics, retrieval confidence — keep it simple until data exists.
- Timeouts shorter than the user's patience; fallback must still be safe (no extra tools).
- Tag
model_idon every span (observability). You cannot debug mix without it. - Prompt versions per model — a v13 prompt on the 8B is not v13 on the 70B.
Provider model names churn. Route to roles (faq_fast, reasoner) bound to ids in config, not to a hardcoded trophy string in the app.
Real-world example
100% frontier model. Cost 8×. Quality +2 on FAQs that a small model already nailed. You route FAQs down, keep the large model for conflicts, eval both slices. Quality holds; invoice drops.
Technical explanation
Uncertainty: if the small model can emit a calibrated "I don't know" or fail schema, escalate. Don't escalate because the prose was short.
Fallbacks must not widen agency (security). A backup that suddenly has refund when the primary didn't is an incident.
Canary: 5% to the new role, kill switch, gold-set gate.
Common mistakes
Always-on dual-model hedge because p50 looked nicer in a demo week.
- Routing on vibes from five chats.
- Fallback that retries infinitely (unbounded consumption).
When to use it
- More than one model in the product, or one model that is too slow/expensive for the easy majority.
When NOT to use it
- Don't add a router before you have an eval slice. Don't route every token through a giant "router model" that costs more than the small path.
Alternatives
- One mid-size model for everything (simpler ops). Distill. Cache (next).
Quick quiz
Related concepts
- Quality, Latency, Cost, and Reliability — Production AI is a tradeoff of quality, latency, cost, and reliability. Optimize dollars per successful task under an SLO — not always the smartest model.
- Latency vs Throughput — Latency is one user's wait (TTFT, ITL, queue); throughput is tokens per second per GPU. Bigger batches raise throughput and hurt latency — pick the SLO first.
Last reviewed: 2026-09-04 · Written by ByHeart AI · Reviewed by ByHeart AI