Design a RAG Chatbot
A RAG chatbot is grounded Q&A: ingest an approved corpus, hybrid-retrieve, rerank, generate only from chunks, cite real ids. It is a chatbot with retrieval — not an agent and not a fine-tune of the handbook.
Explain like I'm new to AI
This is the default "ask our docs" product. The RAG category taught the pieces. Here they sit in the design loop.
docs → chunk → index → hybrid+rerank → grounded answer + citations
Hybrid search + rerank. k small enough to fit the window. Empty retrieve → 'I don't know,' not a fluent guess.
Job: answer questions using our sources, or say we don't know. SLO: faithfulness high, p95 a couple of seconds, cheap enough for FAQ volume.
Shape: chatbot (maybe a tiny workflow: rewrite query → retrieve → generate). Not an agent unless the user must do something.
Knowledge: the index. Update docs, not weights. Fine-tune later only if format still slips.
Mental model
An open-book exam with a bibliography. If the book isn't on the desk, the student must pass.
How it works
- Ingest: chunk, embed, metadata (source, date). Recrawl on a schedule.
- Query: optional rewrite; hybrid + rerank (those lessons).
- Generate: "only from context"; citations the UI can open.
- Eval: context precision/recall vs faithfulness (RAG evaluation). Gold questions from real users.
- Security: retrieved text is untrusted; no send-email on summarize-URL turns; encode HTML.
- Serve: prefix-cache the system prompt; route easy FAQs to a small model.
Empty retrieval is a success path ("I don't know"), not a quality fail.
Real-world example
Internal handbook bot. 120 gold questions. Hybrid search lifted recall; the generator was fine. They almost fine-tuned. They needed rerank, not LoRA.
Technical explanation
Long-context "paste the wiki" dies at corpus scale (lost-in-the-middle, cost). RAG scales. Citations without id verification are theater.
Prompt version + index version on traces so a bad ingest is visible.
Common mistakes
Calling it RAG but only embedding the query and taking top-3 cosine — no hybrid, no rerank, no gold chunks.
- Fine-tuning PDFs into the model.
- Averaging one "quality" score so you never see retrieval was the hole.
When to use it
- Q&A over a corpus you can approve and update. The usual first production AI system.
When NOT to use it
- The user needs multi-step actions (that's support/coding agents). Tiny static 20-page packet → maybe long context instead.
Alternatives
- Search UI without generation (next: AI search). Agent with tools if they must act. Fine-tune for voice after RAG is solid.
Quick quiz
Related concepts
- RAG Architecture — RAG has two phases — offline ingestion (chunk, embed, store) and per-query answering (retrieve, rerank, generate) — connected by a vector store.
- How to Evaluate RAG — RAG evaluation measures retrieval quality (context precision/recall) and generation quality (faithfulness, answer relevance) separately to find and fix failures.
- What is Hybrid Search? — Hybrid search combines dense (semantic) and sparse (keyword) retrieval and fuses their rankings, getting meaning-based recall plus exact-term precision.
Last reviewed: 2026-09-04 · Written by ByHeart AI · Reviewed by ByHeart AI