Reranking The Cheap Win You're Skipping

A cross-encoder reranker is the highest-ROI upgrade in RAG, and most teams skip it. The win is large, the cost is small, and the implementation is a weekend.

Retrieval — BM25, vectors, hybrid — is fast and approximate. Reranking is slower and precise. The two-stage pattern is: retrieve a wide top-K cheaply, rerank the top-K precisely. Most teams never add the second stage and leave quality on the table.

The delta is real. A cross-encoder reranker on top of hybrid retrieval typically lifts retrieval precision materially — the exact number depends on your data, which is why you eval, but the direction is reliable. The cost is one extra model call per query on a small top-K.

• Retrieve top-K (cheap, approximate).
• Rerank top-K (slower, precise).
• Send the reranked top-N to generation.

Reranking is the saw-sharpening that compounds: a small, repeatable upgrade that improves every query from the moment it ships. It's the rare RAG change that's cheap, low-risk, and visible — and that's why it's the first upgrade to make after hybrid search.

Reranking must be evaled against the unrerranked baseline on the same eval set. The precision delta is the case — and on most real data, it's the largest single upgrade a RAG pipeline can make without changing the model.

• Latency: the rerank step adds time; measure it on the top-K, not the whole set.
• Cost: one extra model call per query on a small top-K is cheap; on a large top-K it isn't.
• The reranker model: a small cross-encoder is usually enough.

Reranking is the cheap win most RAG pipelines skip. A cross-encoder on top of hybrid retrieval is the highest-ROI upgrade — cheap, low-risk, and visible on the eval. Add it after hybrid, before anything fancier.