Reliable Decision Making via Calibration Oriented Retrieval Augmented Generation

TL;DR

CalibRAG addresses the problem that LLM-guided decision-making can be poorly calibrated when relying on retrieved content. It introduces a forecasting function f(t, q, d) that produces calibrated confidence signals, trained via synthetic supervision from a surrogate user and integrated into a RAG pipeline with temperature-aware conditioning. Empirical results across general-domain and domain-specific datasets show substantial improvements in calibration (ECE, Brier/NLL) and accuracy over uncertainty calibration baselines, reranking approaches, and robust RAG methods, with ablations confirming the value of temperature conditioning and query reformulation. The approach enables safer, more trustworthy decision support in practice, at the cost of synthetic data generation and added inference steps that can be tuned via Stage 3 reformulation and the epsilon threshold.

Abstract

Recently, Large Language Models (LLMs) have been increasingly used to support various decision-making tasks, assisting humans in making informed decisions. However, when LLMs confidently provide incorrect information, it can lead humans to make suboptimal decisions. To prevent LLMs from generating incorrect information on topics they are unsure of and to improve the accuracy of generated content, prior works have proposed Retrieval Augmented Generation (RAG), where external documents are referenced to generate responses. However, previous RAG methods focus only on retrieving documents most relevant to the input query, without specifically aiming to ensure that the human user's decisions are well-calibrated. To address this limitation, we propose a novel retrieval method called Calibrated Retrieval-Augmented Generation (CalibRAG), which ensures that decisions informed by RAG are well-calibrated. Then we empirically validate that CalibRAG improves calibration performance as well as accuracy, compared to other baselines across various datasets.

Figures (16)

• Figure 1: (a) Cumulative accuracy with the top-K documents on our synthetic validation set (see \ref{['sec:data']}).contriever‑msmarco gains 11% compared to top‑1 when the top‑9 documents are used, showing that the top‑1 hit is often not optimal. CalibRAG reaches a higher top‑1 accuracy and gains little from additional documents. (b, c) Reliability diagrams on NaturalQA. For Llama‑3.1‑8B trained under the Number baseline (see \ref{['sec:exp:set']}), adding the retrieved document (c) raises accuracy relative to the no‑document baseline (b) but also increases ECE, indicating greater over‑confidence. Bar height is the mean accuracy in each confidence bin; darker shading marks bins with more predictions.
• Figure 2: Comparison between CalibRAG and other reranking methods during inference. In contrast to conventional methods that rely on relevance scores to rerank retrieved documents, CalibRAG leverages a confidence score derived from the user's risk tolerance $t$ to guide the reranking process.
• Figure 3: Evaluation results of the baselines and CalibRAG using two retriever models: BM25 and Contriever on NQ and WebQA. We report four metrics—1-AUROC, 1-ACC, ECE, and BS—where lower values indicate better performance.
• Figure 4: Evaluation results of the baselines and CalibRAG utilizing MedCPT on BioASQ-Y/N, MMLU-Med, and PubMedQA. We report four metrics—1-AUROC, 1-ACC, ECE, and Brier Score—where lower values indicate better performance.
• Figure 5: (a) Calibration with and without temperature conditioning on the NQ dataset using Contriever. (b) Effect of the number of retrieved documents on reranking performance on the WebQA dataset using BM25. (c) Impact of query reformulation during inference.

Theorems & Definitions (1)

• Remark