ASRank: Zero-Shot Re-Ranking with Answer Scent for Document Retrieval

TL;DR

This work tackles the sensitivity of RAG systems to the quality of top retrieved documents by introducing ASRank, a zero-shot re-ranking method based on answer scent. A large language model generates an answer scent once, which a smaller ranker then uses to evaluate and re-order retrieved passages via cross-attention and a Bayes-inspired scoring rule. Across diverse open-domain, entity-centric, temporal, multi-hop, BEIR, and TREC datasets, ASRank yields substantial Top-1 improvements over strong baselines and is notably more cost-efficient than query-intensive RankGPT-based approaches. The results demonstrate significant practical gains for ODQA and related retrieval tasks, with concrete reductions in latency and robust improvements in end-to-end RAG performance, while also outlining limitations related to scent quality and dependency on the initial retrieval stage.

Abstract

Retrieval-Augmented Generation (RAG) models have drawn considerable attention in modern open-domain question answering. The effectiveness of RAG depends on the quality of the top retrieved documents. However, conventional retrieval methods sometimes fail to rank the most relevant documents at the top. In this paper, we introduce ASRank, a new re-ranking method based on scoring retrieved documents using zero-shot answer scent which relies on a pre-trained large language model to compute the likelihood of the document-derived answers aligning with the answer scent. Our approach demonstrates marked improvements across several datasets, including NQ, TriviaQA, WebQA, ArchivalQA, HotpotQA, and Entity Questions. Notably, ASRank increases Top-1 retrieval accuracy on NQ from $19.2\%$ to $46.5\%$ for MSS and $22.1\%$ to $47.3\%$ for BM25. It also shows strong retrieval performance on several datasets compared to state-of-the-art methods (47.3 Top-1 by ASRank vs 35.4 by UPR by BM25).

Figures (5)

• Figure 1: After re-ranking the top 1,000 passages retrieved by DPR karpukhin-etal-2020-dense with ASRank, our method surpasses the performance of strong unsupervised models like UPR sachan2022improving on the Natural Questions and TriviaQA datasets.
• Figure 2: Our ASRank framework, starts with document retrieval, re-ranking using the answer scent from LLMs, and finally passing the top-k document into the RAG system.
• Figure 3: Effect of the number of passage candidates on the accuracy of Top-1, 5, 10 results, and latency when re-ranked with LLama 8B and GPT 175. The results were computed on the NQ development set using BM25 retrieved passages.
• Figure 4: Impact of target answer length on ranking performance using DPR on NQ-test.
• Figure 5: Comparison of Exact Match (EM) scores across three datasets (NQ, TriviaQA, and WebQA) for various retrieval models.