Offline vs Online Evaluation
Offline evaluation grades a frozen set in the lab (CI, PRs). Online evaluation watches live users. You need both: offline so you don't ship a regression, online because the real questions are never exactly your gold set.
Explain like I'm new to AI
Offline is a dress rehearsal: the same 120 tickets every time, scored by code, a rubric, or a judge. Fast, cheap, nobody is hurt.
Online is opening night: thumbs, "did the ticket close?", p95 latency, cost per session, A/B tests. Honest about the real mix of questions — and slow, noisy, and able to hurt people.
When: Before users see it — CI, PRs, nightly jobs.
What you score: A frozen set of questions + gold answers (or a rubric). Score exact match, schema, retrieval, or a judge.
Why you need it: Cheap, repeatable, nobody gets a bad answer. Catches regressions the day you change a prompt.
Blind spot: The set is not the real world. Overfit it and you'll look great while production quietly fails.
2026 default: evals as code. A prompt change is a PR. CI runs the gold set. If faithfulness drops, the PR fails. Then you shadow the new prompt on a slice of traffic (score it, don't show it) and only then A/B.
Mental model
Crash tests vs real roads. You do not skip the crash test because "we'll see how customers drive." You also do not ship a car that only survived the lab if the lab never included rain.
How it works
Offline
- Gold questions, gold answers and/or gold retrieved chunks, plus unit checks (JSON schema, tool name, citation id exists).
- Run on every change. Store traces (inputs, outputs, scores) so you can debug a drop.
- Refresh the set when production queries shift — quarterly is common; after a new product line, immediately.
Online
- Task success (refund issued correctly), CSAT/thumbs, human review samples, latency, cost, error rate.
- A/B only after offline pass. High variance: you need more traffic than a button color test.
- Shadow / canary: new system scores in parallel; users still see the old one until the delta is boring.
Real-world example
Offline: new chunker lifts context recall +11 on the gold set. Online: a new mobile FAQ nobody labeled appears; recall looks fine in CI and users still get empty answers. You add those FAQs to the gold set. That loop is the job.
Technical explanation
Offline overfits if you tune prompts against the same 80 items for months (Goodhart: the measure becomes the target). Online confounds season, UI, and model-provider drift.
Replay production traces offline (strip PII): yesterday's real queries become tomorrow's lab. That is the bridge — not a bigger public benchmark.
Do not A/B a prompt you never gated. Do not trust a 99% gold-set score if you never sample live thumbs.
Common mistakes
Only online thumbs. You will learn slowly, hurt users, and never know which component failed.
- Only offline: a beautiful lab score on last quarter's tickets.
- Calling a 200-chat A/B "statistically significant" on a noisy LLM.
When to use it
- Offline: every PR that can change answers.
- Online: after the gate, continuously, with a kill switch.
When NOT to use it
- Don't skip offline "because we'll A/B." Don't skip online "because CI is green."
Alternatives
- Shadow eval is the compromise when A/B is expensive. Human spot-checks of live traces when you cannot instrument task success yet.
Quick quiz
Related concepts
- Benchmarks, Gold Sets, and Regression Tests — Public leaderboards pick models; your private gold set and CI tests pick whether your product got worse. Never train on the eval set.
- Quality, Latency, Cost, and Reliability — Production AI is a tradeoff of quality, latency, cost, and reliability. Optimize dollars per successful task under an SLO — not always the smartest model.
- LLM-as-a-Judge — LLM-as-a-judge grades outputs against a rubric at scale. Calibrate it on humans, and watch verbosity, position, and self-preference bias.
- What is AI Observability? — AI observability reconstructs one request — model, tokens, retrieve, tools — not a green dashboard. Eval scores the product; traces explain a failure.
- Prompt Versioning and Evaluation Traces — Tag every span with prompt, model, and retriever versions. Eval traces match prod shape so you can replay an incident, then flip the flag.
Last reviewed: 2026-09-04 · Written by ByHeart AI · Reviewed by ByHeart AI