Beginner8 min read

What is RAG?

RAG lets a model answer using information it was never trained on — by looking it up first.

Explain like I'm new to AI

Imagine you ask a very smart friend a question about your company's internal handbook. Your friend is brilliant, but they've never read your handbook, so they might guess — and guess wrong.

Now imagine that just before answering, your friend quickly flips to the exact pages of the handbook that matter, reads them, and then answers using those pages.

That's RAG — Retrieval-Augmented Generation. The "friend" is a language model. The "flipping to the right pages" is retrieval. The model still writes the answer, but now it's grounded in real information you handed it.

DocumentYour source data

ChunkSplit into passages

EmbedTurn into vectors

RetrieveFind similar chunks

RerankOrder by relevance

GenerateAnswer with context

The RAG pipeline: document → chunk → embed → retrieve → rerank → generate.

Mental model

Picture the model as an open-book exam taker instead of a closed-book one.

Closed book (plain prompting): the model answers only from what it memorized during training.
Open book (RAG): you slip the relevant page into the exam right before the question, and the model reads it before answering.

RAG does not change what the model knows permanently. It changes what the model can see at the moment it answers.

How it works

A typical RAG system has two phases.

1. Ingestion (done ahead of time):

Take your documents (PDFs, web pages, notes).
Chunk them into smaller passages.
Convert each chunk into an embedding (a vector that captures meaning).
Store those vectors in a database you can search.

2. Answering (done per question):

Turn the user's question into an embedding.
Retrieve the most similar chunks from the database.
Optionally rerank them so the best ones come first.
Put those chunks into the model's prompt as context.
The model generates an answer grounded in that context.

Real-world example

You build a support assistant for a product. A user asks:

"How do I reset my password?"

Without RAG, the model might invent steps that don't match your product. With RAG, the system retrieves your actual help article on password resets, hands it to the model, and the model replies with the correct, product-specific steps — and can even cite the article.

Technical explanation

RAG augments a language model's context window with retrieved evidence rather than modifying model weights. Retrieval is usually semantic: both the query and the stored chunks are represented as vectors, and similarity (often cosine similarity) selects the top-k most relevant chunks.

The retrieved chunks are concatenated into the prompt (the "context"), and the model conditions its generation on them. Because the knowledge lives outside the model, you can update it instantly by changing the underlying documents — no retraining required.

Code

A minimal sketch of the answering phase:

rag.py

# 1. Embed the user's question
query_vector = embed(question)
 
# 2. Retrieve the most similar chunks
chunks = vector_db.search(query_vector, top_k=4)
 
# 3. Build the prompt with retrieved context
context = "\n\n".join(c.text for c in chunks)
prompt = f"""Answer using ONLY the context below.
 
Context:
{context}
 
Question: {question}
"""
 
# 4. Generate a grounded answer
answer = llm.generate(prompt)

Note

The model is told to answer only from the provided context. This is what turns a general chatbot into a grounded, source-backed assistant.

Common mistakes

Common mistake

Thinking RAG "trains" or "fine-tunes" the model. It does neither — it only supplies information at answer time. If you need the model to learn a new skill or style, that's fine-tuning, not RAG.

Retrieving too much and burying the useful chunk in noise.
Chunks that are too big or too small, hurting retrieval quality.
Not evaluating retrieval — if the wrong chunks come back, even a great model gives wrong answers.

When to use it

Your knowledge changes often (docs, policies, catalogs).
You need answers grounded in specific, private, or up-to-date sources.
You want citations so users can verify answers.

When NOT to use it

The knowledge is already in the model and rarely changes.
You need a new behavior or style rather than new facts (use fine-tuning).
The whole knowledge base is tiny and fits comfortably in the context window already.

Alternatives

Fine-tuning — bake knowledge/behavior into the model's weights.
Long context — just paste everything into a very large context window.

Comparison

	RAG	Fine-tuning
Changes the model?	No — supplies context at answer time	Yes — updates weights
Update knowledge	Instant (edit documents)	Requires retraining
Best for	Changing facts, private data, citations	New skills, style, format
Cost to update	Low	Higher

Quick quiz

Related concepts

RAG Architecture — RAG has two phases — offline ingestion (chunk, embed, store) and per-query answering (retrieve, rerank, generate) — connected by a vector store.
RAG vs Fine-tuning vs Long Context — RAG adds knowledge at answer time, fine-tuning bakes behavior into weights, and long context pastes everything into the prompt — each fits different problems.
What is Hybrid Search? — Hybrid search combines dense (semantic) and sparse (keyword) retrieval and fuses their rankings, getting meaning-based recall plus exact-term precision.
Long-term Memory — Long-term memory stores distilled facts outside the model so a new session can retrieve them — the weights never learned the fact.
Multimodal Embeddings & RAG — Multimodal RAG retrieves images, frames, OCR, and audio — keep native media when pixels or sound are the answer.
Design a RAG Chatbot — A RAG chatbot retrieves an approved corpus, reranks, and answers with real citations — not an agent, not a fine-tuned wiki.
Design an AI Search Engine — AI search is query, hybrid retrieve, rerank, then grounded snippets — a chatty generator without ranking metrics is not search.