Quality, Latency, Cost, and Reliability
A production AI system is not "max quality." It is quality, latency, cost, and reliability on a frontier: pick the point that hits the job's SLO, then spend dollars per successful task — not per token of a trophy model.
Explain like I'm new to AI
The smartest model is often the wrong one: too slow, too expensive, flaky at 2 a.m. Evaluation's last job is to make that tradeoff visible.
Floor on faithfulness (eval gate). Then squeeze $ per resolved ticket — router, cache, small model on easy asks.
Four knobs:
- Quality — your gold-set / task-success score (everything earlier in this category).
- Latency — time to first token and time to a finished answer (p50 / p95, not just the mean).
- Cost — dollars per successful task, including retries, tools, RAG, and the judge you run in CI.
- Reliability — timeouts, empty retrieval, provider 500s, fallback actually firing.
You cannot max all four. Chat wants speed. Overnight research wants quality. Support wants a floor on faithfulness then cheap volume.
Mental model
A delivery promise: next-day vs same-hour courier. You pick a service level, then optimize cost inside that promise — you don't buy a fighter jet for pizza.
How it works
- Write SLOs: e.g. p95 under 2s, faithfulness ≥ 0.88 on gold, error rate under 1%, budget $X / resolved ticket.
- Put quality and latency and cost on the same dashboard (online eval). A quality win that blows p95 is a product fail.
- Easy asks → small/fast model or cache. Hard asks → stronger model or tools (routing). Eval both slices.
- Count retries and agent loops in cost. A cheap model that loops eight times is not cheap.
- Reliability: timeouts, fallbacks, "degraded but honest" answers. Eval the fallback path, not only the happy path.
Next category — observability — is how you watch these knobs live. This lesson is why they belong in the same sentence as quality.
Real-world example
Switching to a larger model lifts gold-set correctness +2 and p95 from 1.4s to 7s. Bounce rate eats the gain. A router: small model on FAQ, large on policy conflicts, keeps the SLO and most of the quality. Eval both routes or you'll only measure the easy slice.
Technical explanation
Optimize $ / successful task, not $ / 1k tokens. Tokens hide tool calls, RAG hits, and judge spend.
TTFT vs total time: chat UX cares about first token; a JSON API cares about complete valid payload.
Reliability is an eval target: inject faults in staging (provider down, empty index) and check the system still refuses safely instead of inventing.
Fine-tune + RAG (previous category) often lowers token cost of a giant system prompt — but only if eval quality holds.
Common mistakes
"We use the best model" as architecture. That is a default, not a design. It fails SLOs and invoices.
- Mean latency (hides the p95 horror).
- Quality-only CI: you shipped a correct 12-second answer into a 2-second UI.
When to use it
- Any production assistant or agent. Write the four knobs before you pick a model.
When NOT to use it
- Don't "optimize cost" by dropping the faithfulness gate. Cheap wrong answers are expensive.
Alternatives
- Batch/offline jobs can ignore TTFT. Interactive chat cannot. The tradeoff is the product, not a universal ranking.
Quick quiz
Related concepts
- Offline vs Online Evaluation — Offline eval is a frozen test set in CI; online eval is live users. Ship with offline gates, confirm in production — real queries always drift.
- When to Fine-tune vs RAG vs Prompting — Try prompting, then RAG for facts, then fine-tune for stubborn behavior — the 2026 default is QLoRA SFT (and DPO) plus retrieval, not weights as a wiki.
- What is AI Observability? — AI observability reconstructs one request — model, tokens, retrieve, tools — not a green dashboard. Eval scores the product; traces explain a failure.
- Token Usage, Latency, and Cost Tracking — Track input, cached, reasoning, and output tokens plus TTFT and total time — then dollars per successful task, with alerts on loops and cache busts.
- Latency vs Throughput — Latency is one user's wait (TTFT, ITL, queue); throughput is tokens per second per GPU. Bigger batches raise throughput and hurt latency — pick the SLO first.
- Model Routing and Fallbacks — Send easy turns to a cheap model, hard turns to a strong one, and failures to a fallback — route with eval slices, not hallway demos, and don't hedge every call.
- What is Inference? — Inference is using a trained model to produce outputs — prefill the prompt, then decode tokens. Chat latency is those two phases plus queue, not "the GPU being on."
- How to Design an AI System — Design an AI system in order — job and SLO, shape, knowledge and tools, eval, security, traces, then serving and cost — and do not start with a trophy model.
Last reviewed: 2026-09-04 · Written by ByHeart AI · Reviewed by ByHeart AI