Adaptive Retrieval Without Self-Knowledge? Bringing Uncertainty Back Home

TL;DR

The paper tackles the efficiency-accuracy trade-off in retrieval-augmented QA by exhaustively comparing 35 adaptive retrieval methods against 27 uncertainty estimation techniques across 6 QA datasets and 10 evaluation metrics. It finds that well-established uncertainty estimation methods often match or surpass complex adaptive pipelines in QA while requiring far fewer LLM and retriever calls, and provide stronger self-knowledge signals. The study reveals notable disjunctions between downstream QA performance and self-knowledge quality, especially under distribution shifts, and shows internal-based UE methods to be more functionally complex than consistency-based or logit-based approaches. These results position uncertainty-based adaptive retrieval as a practical, scalable alternative for robust QA with efficient resource use, while highlighting areas for further cross-model validation and ethical considerations.

Abstract

Retrieval Augmented Generation (RAG) improves correctness of Question Answering (QA) and addresses hallucinations in Large Language Models (LLMs), yet greatly increase computational costs. Besides, RAG is not always needed as may introduce irrelevant information. Recent adaptive retrieval methods integrate LLMs' intrinsic knowledge with external information appealing to LLM self-knowledge, but they often neglect efficiency evaluations and comparisons with uncertainty estimation techniques. We bridge this gap by conducting a comprehensive analysis of 35 adaptive retrieval methods, including 8 recent approaches and 27 uncertainty estimation techniques, across 6 datasets using 10 metrics for QA performance, self-knowledge, and efficiency. Our findings show that uncertainty estimation techniques often outperform complex pipelines in terms of efficiency and self-knowledge, while maintaining comparable QA performance.

Figures (16)

• Figure 1: Performance comparison of the state-of-the art models across efficiency metrics (number of LLM calls, Retrieval calls), QA quality metric (In-Accuracy), and the ability to identify self-knowledge (ROCAUC). The plot demonstrates the reverted ranks of the methods across 6 datasets.
• Figure 2: Average overconfidence and underconfidence for each method. Deviation from the zero value is undesirable and indicates erroneous behavior. High OverConfidence values reflect cases where the method incorrectly assumes the model has the required knowledge when it does not. High UnderConfidence values indicate instances where the method fails to recognize that the model already possesses the required knowledge.
• Figure 3: Uncertainty methods average ranks for In-Accuracy, ROC-AUC and Retrieval Calls. Smaller rank indicate average better performance. The In-Accuracy ranks demonstrate key downstream metrics, while the ROC-AUC ranks show self-knowledge abilities across different methods, affecting average downstream performance. The Retriever Calls (RC) ranks represent the efficiency of the method. This evaluation led to choose EigValLaplacian, Lex-Similarity, Max Entropy, Mean Entropy, and SAR for more detailed analysis.
• Figure 4: The transferability of methods between datasets was evaluated using average changes in metrics for Out-Of-Distribution (OOD) data. QA Performance in OOD was measured by InAccuracy, showing comparable results across methods. Self-Knowledge, evaluated by Accuracy, degraded significantly. Efficiency was assessed by RC, indicating that methods tend to call the retriever more frequently after transfer.
• Figure 5: Average loss landscape sharpness in logarithmic scale. Higher values correspond to more complex functions.