RAG Architecture
A RAG system has two phases: an offline ingestion pipeline that makes your documents searchable, and a per-query pipeline that retrieves the right chunks and grounds the model's answer.
Explain like I'm new to AI
RAG isn't one step — it's a small assembly line with two halves that run at different times:
- Ingestion happens ahead of time: you prepare your documents once so they can be searched.
- Answering happens every time a user asks a question: you find the right pieces and hand them to the model.
Understanding these two phases (and where things go wrong in each) is the key to building RAG that actually works.
Mental model
Think of a library. First, a librarian catalogs every book so it can be found later (ingestion). Then, when you ask a question, they fetch the most relevant books, pick the best passages, and summarize them for you (answering). Same library, two very different jobs.
How it works
Phase 1 — Ingestion (offline):
- Parse & clean raw documents (PDFs, HTML, docs) into text.
- Chunk the text into passages, attaching metadata (source, title, date).
- Embed each chunk into a vector.
- Store vectors + metadata in a vector database (build the index).
Phase 2 — Answering (per query):
- Transform the query if needed (rewrite/expand).
- Retrieve the top-k chunks (vector + often keyword = hybrid), applying metadata filters.
- Rerank the shortlist so the best chunks come first.
- Construct the context and prompt the model to answer only from it, with citations.
- Generate the grounded answer.
Real-world example
A company knowledge assistant ingests thousands of internal docs overnight (chunk → embed → store). During the day, each employee question triggers the answering pipeline: retrieve the few relevant chunks, rerank, and generate a cited answer. Update a doc, re-ingest just that doc, and the assistant is instantly current — no model retraining.
Technical explanation
The vector store is the seam between phases. Most failures trace to one side:
- Retrieval failures (wrong chunks) → fix chunking, embeddings, hybrid search, filters, reranking.
- Generation failures (right chunks, wrong answer) → fix the prompt, context construction, grounding instructions, or citations.
That's why you evaluate the two halves separately (retrieval metrics vs. generation metrics). Production systems add caching, observability/tracing, and re-ingestion pipelines to keep the index in sync with source data. Frameworks like LangChain and LlamaIndex provide building blocks, but the architecture above is framework-agnostic.
Common mistakes
Treating RAG as a single black box. When answers are bad, you must know which phase failed — did retrieval bring the wrong chunks, or did the model ignore the right ones? Debug each phase separately.
- Skipping metadata at ingestion, then being unable to filter or cite later.
- Never re-ingesting, so the index drifts out of date with the source documents.
When to use it
- Any non-trivial RAG system — the two-phase structure is the standard blueprint.
When NOT to use it
- Tiny, static knowledge that already fits in the context window needs no ingestion pipeline.
Alternatives
- Long-context "stuff everything in the prompt" for small corpora; fine-tuning for behavior/style.
Quick quiz
Related concepts
- What is RAG? — RAG retrieves relevant external information and gives it to a language model as context before it answers.
- What is Chunking? — Chunking splits documents into smaller passages so they can be embedded and retrieved precisely; chunk size and overlap are key tuning knobs for RAG quality.
- Reranking & Cross-Encoders — A reranker re-scores the shortlist of retrieved chunks by reading each query-document pair closely, pushing the most relevant results to the top.
- Design a RAG Chatbot — A RAG chatbot retrieves an approved corpus, reranks, and answers with real citations — not an agent, not a fine-tuned wiki.
Last reviewed: 2026-09-01 · Written by ByHeart AI · Reviewed by ByHeart AI