Why You Should Always Use a Reranker When Doing RAG

If you’re implementing retrieval augmented generation (RAG), there’s one crucial component you might be missing: a reranking model. While vector similarity search has become the go-to method for retrieving relevant context, relying solely on similarity scores can lead to suboptimal results. Let me show you why reranking is not just an optional enhancement, but a necessary component of any robust RAG system.

The Problem with Pure Similarity Search

When you perform a vector similarity search, you typically specify how many results you want returned–let’s say 5. The database dutifully returns the top 5 most similar results based on vector embeddings. However, there’s a catch: while some results might be highly relevant to your query, others could be only tangentially related. The problem is that you have no reliable way to distinguish between them based on similarity scores alone.

What is a Reranking Model?

Before diving into our example, let’s understand what a reranking model actually is. A reranking model is a specialized machine learning model designed to do one thing really well: determine how relevant a piece of text is to a given query. Unlike embedding models that convert text into vectors for similarity search, reranking models directly compare the query and potential results to assign relevance scores.

Think of it this way:

• An embedding model says “these texts use similar words and concepts”
• A reranking model says “this text actually answers the question being asked”

Reranking models are typically trained on massive datasets of queries and relevant/irrelevant results, learning to understand the subtle relationships between questions and answers. They can catch nuances that simple vector similarity might miss, like whether a document about “API keys in S3” is actually relevant to a question about “API keys in Elasticsearch,” even though both texts discuss API keys.

A Real-World Example

Let’s look at a concrete example from a documentation search system. Imagine a user asks about configuring an API key for Elasticsearch. The vector similarity search returns these results:

Source	Similarity	Relevance
elastic-quickstart	0.72417	0.97080
elastic/integration	0.73912	0.95498
elastic/setup	0.71257	0.91086
s3-bucket-setup	0.71880	0.00986
couchbase-quickstart	0.71165	0.00007

Results from RAG pipeline using OpenAI (embeddings), Pinecone (DB), and Cohere (reranker)

Notice something interesting? Looking at the similarity scores alone, all results appear roughly equivalent, ranging from 0.71 to 0.74. You might think they’re all equally relevant. But they’re not–and this is where reranking shows its true value.

The Power of Relevance Scores

Look at the relevance scores produced by the reranking model:

• The Elasticsearch-related documents score between 0.91 and 0.97
• The S3 and Couchbase documents score below 0.01

The difference is stark and meaningful. The reranking model clearly identifies which documents are truly relevant to the query about Elasticsearch API keys, and which ones just happen to have similar vector embeddings because they also discuss API keys, but in completely different contexts.

Why This Matters for Your LLM

When you feed irrelevant context to a large language model (LLM), even if it’s only somewhat similar to the query, you risk:

1. Getting responses that mix information from different contexts

2. Generating confused or misleading answers

3. Wasting tokens on irrelevant information

In our example, without reranking, the LLM might start discussing S3 bucket API keys or Couchbase authentication when the user only cares about Elasticsearch. By using a reranker and filtering out results below a certain relevance threshold (e.g., 0.5), you ensure that only truly relevant context reaches your LLM.

Implementing Reranking in Your RAG Pipeline

Adding a reranking step to your RAG pipeline is straightforward:

1. Perform your initial vector similarity search
2. Pass the results through a reranking model
3. Filter out results below your chosen relevance threshold
4. Send only the highly relevant context to your LLM

You can implement this in your AI application code, or you can use a RAG pipeline platform like Vectorize which has built-in support for reranking. If you use the Vectorize retrieval endpoint, you can set `rerank=true` and get back both similarity and relevance scores from your similarity search.

Conclusion

The evidence is clear: if you’re building a RAG system, you should absolutely include a reranking step. Similarity scores alone aren’t enough to guarantee relevant context for your LLM. Reranking provides the crucial quality filter that helps your RAG system deliver accurate, focused responses. Don’t let irrelevant context compromise your LLM’s performance–implement reranking today.