Reranking & Cross-Encoders
Reranking takes the shortlist from fast retrieval and re-scores it with a slower, sharper model that reads each query-document pair — lifting the truly best chunks to the top.
Explain like I'm new to AI
Vector search is fast but coarse: it compares pre-computed embeddings, so the perfect answer sometimes lands at #3 or #7 instead of #1. Feed those slightly-wrong top results to the model and answers suffer.
A reranker fixes the order. It takes the top ~20–100 retrieved chunks and re-scores each one by actually reading the query and chunk together, then reorders. Watch it work:
Query: "how do I reset my password?" — first by vector similarity, then reranked by a cross-encoder.
- 1Login troubleshooting overview
- 2Account security best practices
- 3Password reset: step-by-step
- 4Billing and refunds
- 5Two-factor authentication setup
Vector similarity is fast but coarse — the ideal answer sits at #3.
Mental model
Retrieval is a quick scan of the shelf that grabs a stack of maybe-relevant books. The reranker is the careful reader who skims each one against your exact question and puts the best on top. Slow to do for the whole library — perfect for a small stack.
How it works — bi-encoder vs cross-encoder
- Bi-encoder (retrieval): embeds the query and each document separately, then compares vectors. Fast (documents pre-embedded), but the two never "see" each other.
- Cross-encoder (reranking): feeds the query and a document together into the model, which outputs a precise relevance score. Much more accurate, but too slow to run over millions of docs.
The winning pattern combines them: retrieve a shortlist with the fast bi-encoder, then rerank that shortlist with the accurate cross-encoder.
Real-world example
A support bot retrieves the top 25 chunks by vector similarity, then a reranker re-scores them and keeps the top 4 for the prompt. The correct "password reset" article, originally ranked #3, jumps to #1 — so the model answers from the right source. This two-stage retrieve-then-rerank flow is standard in strong RAG systems.
Technical explanation
Rerankers are cross-encoders (or increasingly LLM-based rerankers) that produce a relevance score per (query, chunk) pair. Because cost scales with the shortlist size, you retrieve wide (high recall) then rerank narrow (high precision) — getting the best of both. Trade-offs: extra latency and cost per query, so tune the shortlist size. Rerankers also help fuse hybrid results into one high-quality ordering and reduce the number of chunks you must stuff into the context window (cheaper, less "lost in the middle").
Common mistakes
Skipping reranking and just sending the top-k vector hits to the model. Vector similarity alone often mis-ranks the best chunk — a reranker is one of the highest-ROI additions to a RAG pipeline.
- Reranking too many chunks (latency blows up) or too few (the best chunk never made the shortlist).
- Confusing retrieval (bi-encoder, fast) with reranking (cross-encoder, accurate).
When to use it
- Almost every production RAG system — especially when precision of the top few chunks matters.
When NOT to use it
- Ultra-low-latency paths where the extra step isn't affordable, or when retrieval is already near-perfect.
Alternatives
- Better embeddings and hybrid search improve first-stage retrieval, but rarely match a reranker's top-end precision.
Quick quiz
Related concepts
- RAG Architecture — RAG has two phases — offline ingestion (chunk, embed, store) and per-query answering (retrieve, rerank, generate) — connected by a vector store.
- What is Hybrid Search? — Hybrid search combines dense (semantic) and sparse (keyword) retrieval and fuses their rankings, getting meaning-based recall plus exact-term precision.
- What is Query Transformation? — Query transformation rewrites, expands, or reframes the user's question before retrieval so the search finds better chunks.
- Design an AI Search Engine — AI search is query, hybrid retrieve, rerank, then grounded snippets — a chatty generator without ranking metrics is not search.
Last reviewed: 2026-09-01 · Written by ByHeart AI · Reviewed by ByHeart AI