Improving Zero-shot LLM Re-Ranker with Risk Minimization

TL;DR

UR^3 introduces a bias-aware, unsupervised re-ranking framework that leverages Bayesian decision theory to jointly optimize query- and document-generation likelihoods in a zero-shot LLM setting. By framing the problem with KL-divergence-based loss terms and reformulating the discrepancy between an LLM’s document distribution and the actual document distribution as a tractable ELBO-based objective, UR^3 achieves higher Top-1 accuracy and improved QA performance using fewer input documents. Across multiple open-domain QA datasets and retrieval backbones, UR^3 consistently outperforms the Unsupervised Passage Re-ranker (UPR) and yields substantial gains at the top ranks, while remaining computationally competitive with existing unsupervised methods. The work highlights practical benefits for RAG systems, particularly when few documents are available, and discusses limitations related to Top-20/50 gains and potential latency with large candidate pools. Overall, UR^3 provides a principled, theoretically grounded improvement to zero-shot re-ranking with impactful implications for real-time QA and information retrieval tasks.

Abstract

In the Retrieval-Augmented Generation (RAG) system, advanced Large Language Models (LLMs) have emerged as effective Query Likelihood Models (QLMs) in an unsupervised way, which re-rank documents based on the probability of generating the query given the content of a document. However, directly prompting LLMs to approximate QLMs inherently is biased, where the estimated distribution might diverge from the actual document-specific distribution. In this study, we introduce a novel framework, $\mathrm{UR^3}$, which leverages Bayesian decision theory to both quantify and mitigate this estimation bias. Specifically, $\mathrm{UR^3}$ reformulates the problem as maximizing the probability of document generation, thereby harmonizing the optimization of query and document generation probabilities under a unified risk minimization objective. Our empirical results indicate that $\mathrm{UR^3}$ significantly enhances re-ranking, particularly in improving the Top-1 accuracy. It benefits the QA tasks by achieving higher accuracy with fewer input documents.

Figures (9)

• Figure 1: Method comparison in the re-ranking task. (a) The framework of LLM-based QLM method: unsupervised passage re-ranker (UPR). (b) The framework of our proposal: Unsupervised Risk-minimization Re-Ranker ($\mathbf{UR^3}$); (b1) calculating document generation probability to quantify the biased model estimation; (b2) calculating the query generation probability to measure relevance.
• Figure 2: The process for a LLM-based re-ranking method in the view of Bayes decision theory.
• Figure 3: Visualization of Analysis on the Enhanced Performance in the Re-ranking task
• Figure 4: Distributed correlation in answer generation with normalized NLL in the QA task.
• Figure 5: Effect of the number of document candidates on Top-1 accuracy and calculation efficiency when re-ranked with LLaMA2-7B model. Evaluation is done on the NQ test set using BM25 retrieved documents.