Retrieve & Re-Rank

In Semantic Search we have shown how to use SparseEncoder to compute embeddings for queries, sentences, and paragraphs and how to use this for semantic search. For complex search tasks, for example question answering retrieval, the search can significantly be improved by using Retrieve & Re-Rank. Note that a detailed explanation with dense embeddings produced by Bi-Encoder is accessible here.

Overview

The Retrieve & Re-Rank approach consists of two stages:

  1. Retrieval Stage: Use fast but less accurate methods (SparseEncoder/bi-encoders) to retrieve a larger set of potentially relevant documents

  2. Re-Ranking Stage: Use more sophisticated but slower models (cross-encoders) to re-rank the retrieved documents for better precision

This approach combines the efficiency of first-stage retrieval with the accuracy of second-stage re-ranking.

Comprehensive Evaluation: Hybrid Search Pipeline

File: hybrid_search.py

This script provides a complete evaluation pipeline comparing different retrieval and re-ranking approaches on a given dataset (here in our example NanoNFCorpus). It includes:

  1. Sparse Retrieval using ibm-granite/granite-embedding-30m-sparse

  2. Dense Retrieval using sentence-transformers/multi-qa-MiniLM-L6-cos-v1

  3. Re-ranking both sparse and dense results with cross-encoder/ms-marco-MiniLM-L6-v2

  4. Hybrid Search using Reciprocal Rank Fusion ReciprocalRankFusionEvaluator

  5. Hybrid Re-ranking applying cross-encoder to fused results

Output: The script generates comprehensive metrics and saves results in the runs/ directory.

Evaluation Results

Example results from running the hybrid search evaluation on NanoNFCorpus. Reranking uses always_rerank_positives=False, so each reranker receives only the documents returned by its retrieval stage. MAP and NDCG use all known positives for normalization.

================================================================================
EVALUATION SUMMARY
================================================================================
METHOD                            NDCG@10     MRR@10        MAP
--------------------------------------------------------------------------------
Sparse Retrieval                    32.20      46.91      14.30
Dense Retrieval                     27.35      41.59       8.95
Sparse + Reranking                  34.30      55.21      14.14
Dense + Reranking                   34.63      55.10      13.63
Hybrid RRF                          33.02      50.19      12.84
Hybrid RRF + Reranking              35.34      55.27      14.61
================================================================================

In this run, reranking improves MRR@10 and NDCG@10 for all three retrieval methods. MAP improves for dense and hybrid retrieval and decreases slightly for sparse retrieval.

Pre-trained Models

Sparse Encoder (Retrieval)

The SparseEncoder produces embeddings independently for your paragraphs and for your search queries. You can use it like this:

from sentence_transformers import SparseEncoder

model = SparseEncoder("naver/splade-cocondenser-ensembledistil")

docs = [
    "My first paragraph. That contains information",
    "Python is a programming language.",
]
document_embeddings = model.encode_document(docs)

query = "What is Python?"
query_embedding = model.encode_query(query)

For pre-trained Sparse Encoder models, see: Pretrained Sparse-Encoders.

Cross-Encoders (Re-Ranker)

For pre-trained Cross Encoder models, see: MS MARCO Cross-Encoders