What is reranking in RAG?

Reranking is a second-stage pass that reorders the chunks your retriever pulled back, scoring each one against the question with a slower but far more accurate model. Your vector search fetches a broad set of candidates in milliseconds; the reranker then reads each candidate together with the query and re-sorts them by true relevance, so the best few land at the top before anything reaches the LLM.

Why is reranking often the highest-ROI quality boost?

First-stage retrieval optimizes for speed and recall — it casts a wide net and usually catches the right chunk somewhere in its top 20 or 50. The problem is that the genuinely best chunk is often sitting at rank 8, not rank 1, and the LLM pays most attention to whatever you put first. A reranker fixes exactly that ordering problem.

For many teams it's the single cheapest change that noticeably improves answers, because you're not re-indexing anything or swapping your embedding model — you're just adding one scoring step at the end.

How does a bi-encoder differ from a cross-encoder?

The two stages use different model shapes on purpose.

Because a cross-encoder sees the query and chunk side by side, it catches nuance that a raw vector distance misses — but it has to run fresh for every candidate, so you only unleash it on the shortlist the bi-encoder already narrowed down.

What does reranking cost you?

Latency and money. Each candidate is an extra model call, so reranking 50 chunks is meaningfully slower and pricier than reranking 10. The usual levers for keeping it in check:

• Retrieve wide (say 30-50 chunks), rerank, then keep only the top 3-5 for the prompt.
• Use a hosted reranker API for simplicity, or a self-hosted open cross-encoder to control cost.
• Skip reranking on trivial lookups and reserve it for queries where precision actually matters.

For most RAG stacks the trade is worth it: a bit more latency in exchange for the right chunk landing first, which is often the difference between a good answer and a wrong one.