RAG Reranking: When and Why to Add a Reranker
By
Tom Dallimore
Published
RAG has a particularly annoying failure mode: the correct answer is sitting right there in the retrieved documents, but it is ranked low enough that the language model never sees it.
Lovely.
Your retriever did technically find the right chunk. It just put three generic overview pages, two stale policies and a vaguely related support article above it. Then your prompt takes the top five results, the useful evidence gets cut off, and everybody blames the LLM.
That is the problem RAG reranking is designed to fix. In a two stage retrieval setup, the first retriever optimizes for speed and retrieval recall. It casts a reasonably wide net using vector search, keyword search or hybrid search. A second model then takes that candidate set and performs a slower, more careful ranking pass before the strongest relevant context reaches the prompt.
The important bit is that reranking is not a magic quality switch. If your RAG system cannot retrieve the relevant documents at all, a reranker has nothing useful to reorder. Fix recall, chunking, stale data and filters first. Add a reranker when the right evidence is already being found but the ranking is the real problem.
This guide explains how re ranking works, where a cross encoder fits, when it is worth the extra latency and cost, how to evaluate it properly, and when you should leave your existing retriever alone.
What Is RAG Reranking?
RAG reranking is a second-stage ranking step inside retrieval augmented generation. The first search returns more documents than you actually want to send to the model. The reranker then scores those candidates against the original user question and moves the most relevant documents to the top.
A simple example looks like this:
Retrieve 40 candidate chunks from a vector database, keyword index or hybrid search layer.
Score those 40 query-document pairs with a stronger reranking model.
Keep the top k results, perhaps the best 5 to 8 chunks.
Build the prompt from that smaller set of retrieved context.
Let the large language model generate the final answer from better evidence.
The first stage is about high recall. The second stage is about precision. You want the initial retrieval to avoid missing useful candidates, then you want the reranker to decide which of those candidates actually deserve space in the context window.
If you want the full plumbing around that process first, my RAG Pipeline guide covers ingestion, chunking, embeddings, retrieval, reranking, prompting and generation end to end.
Where Reranking Fits in the RAG Pipeline
A normal RAG pipeline already has a ranking step, even if you have never explicitly added a reranker.
Your embedding model converts a query into a vector. Vector search then compares that embedding with stored document embeddings using vector similarity, often with cosine similarity or another semantic similarity metric. Keyword retrieval may run alongside it. Metadata filters may shrink the search space. Then the search engine returns a ranked list.
That first ranking is fast, but it is not always nuanced enough.
The reranking stage sits between retrieval and generation:
User question → Retrieve candidates → Rerank → Top context → LLM → Answer
A bi encoder can search millions of chunks quickly because it encodes the query and documents separately. That is perfect for broad retrieval. A cross encoder is much slower because it reads the query and each candidate together, but that joint comparison produces a more useful relevance score for difficult ranking tasks.
So the architecture is deliberately asymmetric: fast model first, stronger second model after the search space has already been reduced.
Why First-Stage Retrieval Gets the Order Wrong
The first retriever is doing a difficult job under a tight latency budget. It is trying to find broadly relevant documents without deeply rereading every candidate for every query.
That creates a few predictable problems:
Dense retrieval can understand meaning but miss the importance of specific terms such as product IDs, version numbers, dates or negation.
Keyword search can nail exact wording but struggle with paraphrases and semantic meaning.
Near-duplicate pages can all look relevant even when only one is current.
A large chunk may score well because part of it is related, even though the actual answer lives in a smaller competing chunk.
Hybrid search can improve the candidate pool while still leaving you with a messy final ordering.
Imagine a user asks for the 2026 enterprise cancellation policy. Your knowledge base contains a 2024 policy, a 2025 policy, a general account-closing article and the current 2026 policy. Vector similarity may decide all four are very relevant. Keyword matching may love them too. The real problem is not retrieval recall because the correct answer is already in the candidate set. The real problem is ranking.
This distinction matters. If the right chunk is at rank 17 and your prompt only receives the top 6, reranking can help. If the right chunk never appears in the top 40, you do not have a reranking problem. You have a retrieval problem.
For that wider diagnosis, start with How to Improve RAG Performance. It covers chunking, hybrid retrieval, query handling, evaluation and the boring upstream failures that a reranker cannot rescue.
Bi-Encoder vs Cross-Encoder: The Useful Difference
The words sound more academic than the idea.
Bi encoder retrieval encodes the query and each document independently. Those embeddings can be stored ahead of time in a vector database, so at runtime you only create the query embedding and compare it against the index. That makes bi encoder retrieval fast enough for large candidate pools.
Cross encoder reranking sends the query and one candidate chunk through the model together. Because the model can inspect both inputs at once, it can reason about finer relationships such as exact intent, wording, contradiction and whether the chunk actually answers the question.
The trade off is obvious: deeper comparison costs more compute.
Stage | Typical job | Strength | Weakness |
Bi encoder | First-stage retrieval | Fast, scalable, high recall | Coarser relevance |
Cross encoder | Second-stage reranking | High precision, better query-document comparison | Slower, cost grows with candidate count |
Late interaction | Balanced retrieval/ranking | Retains more token-level detail | More operational complexity |
Larger models / LLM rerankers | Complex low-volume cases | Flexible reasoning | Expensive and slower |
Common pre trained models for cross encoder reranking include models from Sentence Transformers, such as cross-encoder/ms-marco-MiniLM-L-6-v2. More advanced techniques use late-interaction models or larger rerankers, but careful tuning matters more than chasing the biggest model. The exact model matters less than testing whether it improves your own ranking.
And if your embeddings are still the weak link, fix that first. My RAG Embeddings guide explains how the embedding model, vector space, similarity metric and real-query evaluation affect first-stage retrieval.
When Should You Add a Reranker?
Do not add a reranker because somebody's architecture diagram had one. Add it because your test set shows a ranking problem.
The strongest signal is simple: retrieval finds the answer, but the answer does not rank high enough.
Retrieval recall is strong, but Precision@1 or MRR is weak.
The right chunk regularly appears in positions 10 to 30, while the top few results are merely adjacent.
Your system retrieves several near-duplicate documents and frequently chooses the wrong version.
User feedback points to an existing document the system did retrieve, just not high enough.
The top results are broadly relevant, but only one actually answers the user question.
Answer quality improves when you manually move a lower-ranked chunk into the provided context.
Those are good reranking problems because the candidates already contain the evidence you need.
When Not to Add a Reranker Yet
This is the more important section.
A reranker will not repair a broken candidate set. It cannot promote a document that the search layer never retrieved.
Relevant documents are missing entirely from the top 20 to 50 candidates.
Chunk boundaries have split the answer away from the context that explains it.
Your knowledge base is stale, duplicated or missing important data.
Metadata or permission filters are excluding the correct documents.
The query needs rewriting because the user's input is vague or depends on previous conversation turns.
You are relying on pure vector search even though exact IDs or specific terms clearly matter.
If exactness is one of the failure modes, use Hybrid RAG before assuming reranking is the only answer. Hybrid search combines semantic search with keyword retrieval, which usually gives the reranker a much healthier candidate set to work with.
If the problem is damaged chunks, use the RAG Chunking Strategies guide first. A cross encoder can compare two pieces of text very carefully. It cannot restore context you removed before indexing.
Designing a Two-Stage Retrieval Pipeline
A sensible two stage retrieval design is deliberately boring:
Run the existing retriever broadly enough to preserve recall.
Merge dense and lexical candidates if you use hybrid search.
Apply permissions, freshness and other hard filters before anything reaches the model.
Send a manageable candidate set to the reranker.
Sort by the new relevance score.
Pass only the strongest top k chunks into the prompt.
Generate the answer and log both the pre-rerank and post-rerank order.
The candidate count is one of the main knobs. You are not trying to retrieve more relevant documents just to increase volume; you are trying to give the second stage enough good candidates to make a useful decision.
Retrieve too few candidates and the reranker has no room to fix the order. Retrieve hundreds for every request and your second stage becomes an expensive queue of model calls. Start with a moderate candidate set, evaluate it, and change the number when the data tells you to.
The same applies to top k. More context is not automatically more relevant context. The goal is not to stuff the context window. It is to provide the language model with enough high-quality evidence to produce the correct answer.
Hybrid Search, Reciprocal Rank and Reranking
Hybrid search and reranking solve different problems, which is why they work well together.
Hybrid search improves candidate generation. Reranking improves the final order of those candidates.
You might run vector search and keyword search in parallel, then combine the two ranked lists using reciprocal rank fusion. Reciprocal rank gives weight to where a result appears in each list instead of pretending a BM25 score and a cosine similarity score are directly comparable.
That merged set becomes the input to the reranker.
Vector search + Keyword search → Reciprocal rank fusion → Candidate set → Cross encoder → Top context
This is especially useful when your domain mixes natural-language questions with exact identifiers. Semantic retrieval finds meaning. Keyword retrieval catches the exact code. The reranker then decides which of those results actually answers the query.
The Vector Databases for RAG guide covers the wider retrieval stack, including vector similarity, metadata filters and hybrid search.
Accuracy, Latency and Cost: The Reranking Trade-Off
Reranking normally improves precision by spending more compute on fewer candidates. That is the whole bargain.
The cost grows with the amount of work you ask the second stage to do. If you double the candidate set, you are roughly doubling the number of query-document comparisons unless your provider or runtime batches them efficiently.
So measure the actual trade off instead of arguing about it in theory:
Retrieval recall before reranking
Precision@1, Precision@k and MRR after reranking
End-to-end answer accuracy and answer quality
Reranker latency
Total request latency
Cost per query or per thousand queries
How often the reranker changes the winning chunk
A 60 ms reranker is basically irrelevant if generation takes four seconds and answer quality improves materially. The same 60 ms can matter a lot in a search-as-you-type experience with a 200 ms latency target.
Context matters more than ideology.
A few practical ways to keep the second stage under control:
Rerank only a moderate number of candidates rather than every retrieved document.
Batch query-document pairs when your model runtime supports it.
Use a smaller cross encoder as the baseline before reaching for larger models.
Cache stable results for repeated queries where that is safe.
Skip reranking for trivial queries if your telemetry shows the first-stage ranking is already excellent.
Keep the reranker modular so you can disable or swap it without rebuilding the RAG pipeline.
How to Evaluate a Reranker Properly
Please do not test this by asking three friendly questions and deciding the new model 'feels better'.
Build a labeled test set from real user questions. For each query, record which document or chunk should rank highly. Include easy questions, awkward wording, specific terms, near-duplicate policies, stale versions, multilingual input if relevant, and questions where several documents look annoyingly similar.
Then compare the same pipeline with reranking on and off.
Did the right chunk move higher?
Did MRR improve?
Did Precision@1 or Precision@k improve?
Did answer accuracy improve when the final answer was generated?
How much latency and cost did the second stage add?
Did the new ranking hurt any important query class?
Keep the embedding model, chunking, filters and candidate count fixed while you compare rerankers. Change five things at once and you will learn absolutely nothing about which change improved quality.
For the broader evaluation setup, including retrieval metrics, ground truth and production regression tests, use the RAG Evaluation guide.
Production Reranking: What to Log
Once reranking is live, log enough to explain what moved.
Original query and any rewritten query
Candidate IDs and scores from the existing retriever
Post-rerank scores and final ranking
Which chunks actually entered the prompt
The generated answer and cited sources
Latency for retrieval, reranking and the model call
Errors, fallbacks and cases where reranking was skipped
This is the difference between knowing 'the answer was bad' and knowing 'the correct document was rank 14 before reranking, moved to rank 2, but the top k cutoff was still 1'. One of those gives you something useful to fix.
It is also the same reason I care about tracing in Fetch Hive. If a RAG workflow fails, I want to see retrieval, ranking, model calls, latency and cost instead of staring at one polished final response and guessing what happened.
Common RAG Reranking Mistakes
Adding a reranker before fixing low recall. If the correct document is absent, there is nothing to rerank.
Reranking a tiny candidate set. Re ranking five already-good results rarely gives the second model much room to help.
Ignoring freshness and permissions. Relevance is not the only ranking signal that matters in production.
Using a huge model by default. Larger models may improve difficult cases, but they can also wreck latency and cost.
Evaluating only retrieval scores. A better ranking should ultimately improve the answer, not just a chart.
Treating one benchmark as universal. Your own documents and user queries are the real test.
Forgetting hybrid retrieval. A reranker is stronger when the candidate pool already contains both semantically relevant and exact-match results.
A Practical Reranking Checklist
Before I added a reranker to a production RAG system, I would want to answer yes to most of these:
Do we have a repeatable test set built from real user queries?
Is retrieval recall already high enough that the relevant documents are usually in the candidate set?
Can we inspect the ranking before and after reranking?
Have we ruled out broken chunking, stale data and bad filters as the real problem?
Do exact terms justify hybrid search before the reranker?
Can we measure Precision@1, MRR, answer accuracy, latency and cost?
Is top k small enough to avoid flooding the LLM with weak context?
Can we disable the second stage easily if it does not improve quality?
If not, that is fine. It just tells you what to instrument before you add another model to the stack.
The Takeaway
Reranking is useful because first-stage retrieval and final ranking are different jobs.
The fast retriever should find a broad, high-recall candidate set. The reranker should look at those candidates more carefully and decide which ones actually answer the query. When the correct document is sitting at rank 17, that second stage can be one of the highest-leverage improvements you make.
But if the correct document never appears, adding a cross encoder is just a more expensive way to sort the wrong results.
Start with retrieval. Fix the data, chunking and search. Use hybrid retrieval where it helps. Measure recall. Then add reranking when your evidence shows that ranking is the real problem.
The goal is not to build the fanciest RAG architecture. The goal is to get the right evidence into the prompt before the model starts talking.
Share this post
