Tools

Reranking

Also known as: Re-ranking, Cross-Encoder Reranking

A second scoring pass in a retrieval pipeline that re-orders an initial set of candidate documents with a more accurate — and more expensive — relevance model. A fast first stage such as vector search or keyword search returns the top few dozen candidates, and the reranker scores each one against the query to produce a better final ordering.

Why it matters

First-stage retrieval is optimized for speed, not precision, so the best chunk often lands at position 12 instead of position 1 — and whatever lands at the top is what the LLM actually sees in its prompt. Reranking buys a large relevance improvement for a modest latency cost, which is why it has become a standard stage in production RAG systems.

Deep Dive

The reason reranking exists is that fast retrieval and accurate retrieval are different problems. The first stage of a RAG or semantic search pipeline has to sift through millions of chunks in milliseconds, so it relies on approximate methods: a bi-encoder embedding model converts the query and every document into vectors independently, or BM25 scores keyword overlap with no understanding of meaning at all. These methods usually get the right document somewhere into the top 50, but the ordering within those 50 is noisy. A reranker takes that candidate set and applies a model that would be far too slow to run against the whole corpus — typically a cross-encoder that reads the query and one document together — and re-sorts the list so the genuinely relevant chunks rise to the top. The final top-k, often just 3–10 chunks, is what gets assembled into the prompt.

Bi-Encoders vs. Cross-Encoders

The performance gap between the two stages comes from architecture. A bi-encoder produces one vector per text: the query is embedded once, each document is embedded once (usually offline, ahead of time), and relevance is just cosine similarity between two points. That is extremely fast with approximate nearest-neighbor indexes, but the model never sees the query and the document together, so it cannot model how specific words in one interact with specific words in the other. A cross-encoder does the opposite: the query and a single candidate document are concatenated and passed through a Transformer as one input, and the model outputs a relevance score directly. Because every token can attend to every token across both texts, it picks up on nuance — negation, exact constraints, multi-hop phrasing — that embedding similarity routinely misses.

The catch is cost. Scoring 50 candidates means 50 separate forward passes, and the pairs cannot be pre-computed because the query is only known at request time. That is why cross-encoders are never run over a full corpus of millions of documents, and why the two-stage split exists in the first place: the bi-encoder's job is recall (get the right documents into the candidate set), and the cross-encoder's job is precision (put them in the right order).

Rerankers in Practice

There are two common ways to add a reranker. Hosted APIs such as Cohere Rerank and Voyage AI's rerank endpoint take a query plus a list of documents in a single request and return scores — no infrastructure to run, but an extra network call and per-request pricing. Open-weight models, led by BAAI's bge-reranker family and Jina AI's rerankers, are available on Hugging Face and can run on a single GPU or even a CPU for small candidate sets, which keeps data in-house and costs predictable.

The integration pattern is the same either way: retrieve a generous candidate set (top 20–100, often from hybrid search over a vector database plus BM25), rerank it, keep the top 3–10, and discard the rest. Some teams skip the dedicated model entirely and prompt an LLM to rank candidates zero-shot; that works, and listwise LLM rerankers can be strong, but a purpose-built cross-encoder is usually one to two orders of magnitude cheaper per query.

It Can't Fix Bad Retrieval

A common misconception is that adding a reranker will rescue a weak search pipeline. It cannot, because reranking only re-orders what the first stage already returned: if the relevant chunk never made the candidate set, the reranker never sees it, and no amount of accurate scoring will bring it back. The first stage sets a hard recall ceiling on everything downstream. This has a practical consequence for debugging: when end-to-end retrieval quality is poor, measure the stages separately. Check whether the right chunk appears anywhere in the top 50 (a retrieval problem — fix embeddings, chunking, or add hybrid search) versus whether it appears but ranks too low (a reranking problem). Teams that skip this distinction often spend weeks tuning a reranker for gains that were capped by stage one all along.

The Latency Budget

Reranking is a trade: better ordering for extra latency and cost. A self-hosted cross-encoder scoring 50 query–document pairs typically adds on the order of tens to a few hundred milliseconds depending on model size and document length; a hosted API adds a network round trip on top of that. The knobs are straightforward: rerank fewer candidates (top 20 instead of top 100), truncate documents before scoring, use a smaller model, or cache scores for repeated queries.

Whether the trade is worth it depends on the application. For a user-facing assistant where answers already take seconds to generate, an extra 100–300 ms is usually invisible, and the relevance gain directly reduces hallucination caused by off-topic context. For autocomplete-style search or high-QPS internal services, teams often reserve reranking for queries that look hard — long, ambiguous, or low-confidence from stage one — and let easy queries go straight through. The pragmatic default in production retrieval systems is: always rerank unless you can prove you cannot afford to.

Related Concepts

← All Terms

← Replicate Resemble AI →