LLM-Confidence Reranker: A Training-Free Approach for Enhancing Retrieval-Augmented Generation Systems

TL;DR

The LLM-Confidence Reranker (LCR) is proposed, a training-free, plug-and-play algorithm that enhances reranking in RAG systems by leveraging black-box LLM confidence derived from Maximum Semantic Cluster Proportion (MSCP).

Abstract

Large language models (LLMs) have revolutionized natural language processing, yet hallucinations in knowledge-intensive tasks remain a critical challenge. Retrieval-augmented generation (RAG) addresses this by integrating external knowledge, but its efficacy depends on accurate document retrieval and ranking. Although existing rerankers demonstrate effectiveness, they frequently necessitate specialized training, impose substantial computational expenses, and fail to fully exploit the semantic capabilities of LLMs, particularly their inherent confidence signals. We propose the LLM-Confidence Reranker (LCR), a training-free, plug-and-play algorithm that enhances reranking in RAG systems by leveraging black-box LLM confidence derived from Maximum Semantic Cluster Proportion (MSCP). LCR employs a two-stage process: confidence assessment via multinomial sampling and clustering, followed by binning and multi-level sorting based on query and document confidence thresholds. This approach prioritizes relevant documents while preserving original rankings for high-confidence queries, ensuring robustness. Evaluated on BEIR and TREC benchmarks with BM25 and Contriever retrievers, LCR--using only 7--9B-parameter pre-trained LLMs--consistently improves NDCG@5 by up to 20.6% across pre-trained LLM and fine-tuned Transformer rerankers, without degradation. Ablation studies validate the hypothesis that LLM confidence positively correlates with document relevance, elucidating LCR's mechanism. LCR offers computational efficiency, parallelism for scalability, and broad compatibility, mitigating hallucinations in applications like medical diagnosis.

Figures (7)

• Figure 1: Illustration of LLM-Confidence Reranker (LCR) algorithm. LCR leverages LLM confidence signals derived from document helpfulness to enhance ranking, distinct from traditional relevance or similarity measures.
• Figure 2: Illustration of Maximum Semantic Cluster Proportion. As illustrated in the figure, the question $q$ is "Who is the President of the U.S.?" We instruct a specific LLM with parameters $\phi$ to sample $K$ times (where $K = 5$ in this example) at a high temperature (typically $T = 1$). Through clustering, three semantic clusters are derived, with the largest semantic cluster being "Trump", which accounts for $3/5$ of the responses, denoted as $\text{MSCP}(q; \phi, K) = 3/5$. When we provide both the question $q$ and the document $d_1$, which states, "Trump was sworn in as the 47th U.S. President on Jan. 20. 2025", all five sampled responses consistently point to "Trump", resulting in $\text{MSCP}(q, d_1; \phi, K) = 5/5$. Compared to the scenario where only $q$ is provided, the MSCP increases, indicating that $d_1$ is a helpful document for $q$. In contrast, when we supply both the question $q$ and the document $d_2$, which claims, "Elon Musk becomes the 47th President of the United States", the largest semantic cluster after sampling and clustering only constitutes $2/5$, denoted as $\text{MSCP}(q, d_2; \phi, K) = 2/5$. This represents a decrease in MSCP compared to when only $q$ is provided, suggesting that $d_2$ is a harmful document for $q$.
• Figure 3: Query Threshold Impact on NDCG@5 for NaturalQuestions with BM25 Retriever. These plots illustrate the impact of varying query threshold (QT) values on performance for different rerankers. The red dashed line indicates the baseline without LCR (QT=0).
• Figure 4: Heatmap of NDCG@5 Improvement Percentages for Document Thresholds. This heatmap shows the percentage differences in NDCG@5 scores for various combinations of the lower threshold (LT, y-axis) and upper threshold (UT, x-axis) in the LCR algorithm, applied to the NaturalQuestions dataset with BM25 as the initial retriever and RankGPT as the reranker, without the query threshold. Red shades indicate positive improvements relative to the baseline, blue shades signify negative changes, and white denotes no change. Color intensities correspond to the magnitude of variations, revealing optimal threshold settings for improved ranking performance.
• Figure 5: NDCG@5 Improvements Across Different Language Models. Results are derived from the Natural Questions dataset.