URAG: A Benchmark for Uncertainty Quantification in Retrieval-Augmented Large Language Models

Abstract

Retrieval-Augmented Generation (RAG) has emerged as a widely adopted approach for enhancing LLMs in scenarios that demand extensive factual knowledge. However, current RAG evaluations concentrate primarily on correctness, which may not fully capture the impact of retrieval on LLM uncertainty and reliability. To bridge this gap, we introduce URAG, a comprehensive benchmark designed to assess the uncertainty of RAG systems across various fields like healthcare, programming, science, math, and general text. By reformulating open-ended generation tasks into multiple-choice question answering, URAG allows for principled uncertainty quantification via conformal prediction. We apply the evaluation pipeline to 8 standard RAG methods, measuring their performance through both accuracy and prediction-set sizes based on LAC and APS metrics. Our analysis shows that (1) accuracy gains often coincide with reduced uncertainty, but this relationship breaks under retrieval noise; (2) simple modular RAG methods tend to offer better accuracy-uncertainty trade-offs than more complex reasoning pipelines; and (3) no single RAG approach is universally reliable across domains. We further show that (4) retrieval depth, parametric knowledge dependence, and exposure to confidence cues can amplify confident errors and hallucinations. Ultimately, URAG establishes a systematic benchmark for analyzing and enhancing the trustworthiness of retrieval-augmented systems. Our code is available on GitHub.

Figures (7)

• Figure 1: Illustration of uncertainty in RAG. When asked "Can pregnant women drink coffee?", the retriever surfaces multiple documents, one stating that "coffee is safe in pregnancy" and another warning that "too much coffee can be risky." The LLM, however, places disproportionate attention on the first document and produces an overconfident yet incomplete answer, overlooking contradictory evidence. This example highlights how noisy or one-sided retrieval can amplify model overconfidence, leading to potentially harmful or misleading responses.
• Figure 2: Illustration of dataset construction and evaluation pipeline for URAG. The upper flow shows how we construct the benchmark, which involves retrieving documents from the database and prompting LLMs to generate a supporting document and wrong answers. After that, we use an NLI model to confirm the difficulty before having annotators check the final result. During the evaluation process, RAG systems decide between a list of wrong answers and the correct answer before measuring performance and uncertainty.
• Figure 3: Accuracy and uncertainty difference under self-aware prompting. When the LLM is exposed to its confidence scores, RAT exhibits the strongest degradation in accuracy and stability, while most other RAG methods show reduced uncertainty, particularly on math (Olympiad) and general-domain tasks (CRAG).
• Figure 4: Accuracy and uncertainty difference under wrong-aware prompting. Misleading confidence cues cause a consistent drop in accuracy and an increase in uncertainty, indicating a higher rate of hallucinated decisions.
• Figure 5: Reverse Correlation between Accuracy and Uncertainty across RAG methods. Each point represents the average accuracy and predictive uncertainty (set size) of one RAG method evaluated on ten datasets. A strong negative correlation ($r=-0.76$) shows that higher accuracy aligns with lower predictive uncertainty across RAG methods.