Rewarding Doubt: A Reinforcement Learning Approach to Calibrated Confidence Expression of Large Language Models

TL;DR

This work addresses the need for trustworthy LLM outputs by calibrating the model’s expressed confidence to match actual correctness. It introduces Rewarding Doubt, an RL framework that jointly tunes answers and the associated confidence using a logarithmic proper scoring rule, ensuring $\hat{p}$ converges to the true $p^*$. The approach integrates confidence expression into generation rather than post-hoc calibration, achieving improved calibration (lower ECE) and discrimination (higher AUROC) on multiple QA benchmarks, with demonstrated generalization to out-of-domain tasks. Practical benefits include minimal inference overhead and stable task performance, supporting safer, more reliable human–AI collaboration in high-stakes settings.

Abstract

A safe and trustworthy use of Large Language Models (LLMs) requires an accurate expression of confidence in their answers. We propose a novel Reinforcement Learning approach that allows to directly fine-tune LLMs to express calibrated confidence estimates alongside their answers to factual questions. Our method optimizes a reward based on the logarithmic scoring rule, explicitly penalizing both over- and under-confidence. This encourages the model to align its confidence estimates with the actual predictive accuracy. The optimal policy under our reward design would result in perfectly calibrated confidence expressions. Unlike prior approaches that decouple confidence estimation from response generation, our method integrates confidence calibration seamlessly into the generative process of the LLM. Empirically, we demonstrate that models trained with our approach exhibit substantially improved calibration and generalize to unseen tasks without further fine-tuning, suggesting the emergence of general confidence awareness. We provide our training and evaluation code in the supplementary and will make it publicly available upon acceptance.

Figures (5)

• Figure 1: Illustration of our reward design: The model is rewarded for a high confidence if it is correct and punished if it is incorrect. To maximize the reward, the model needs to learn when to predict a higher or lower confidence, considering a possible higher punishment. Our reward function is designed so that the best reward is given when the confidence and the epistemic probability of being correct are the same, thus incentivizing the expression of calibrated confidences.
• Figure 2: Overview of our reinforcement learning framework: The LLM is prompted to answer a question and provide the confidence in this answer. The answer is checked for correctness by a judge function and the reward is calculated based on the correctness and the confidence. Correct answers with high confidences are rewarded highly, but also penalized heavily when incorrect.
• Figure 3: The rewards for each confidence value for correct and incorrect answers. The closer the confidence is to ten or zero, respectively, the higher is the reward. At the same time, the possible punishment increases to a greater extent. The model has to learn when the trade-off between those two possibilities is worthwhile.
• Figure 4: Calibration curves of the zero-shot base model (Verbalize) and the model fine-tuned by Rewarding Doubt.
• Figure 5: Histograms of predicted confidences of the zero-shot base model (Verbalize) and the model fine-tuned on the TriviaQA dataset.

Theorems & Definitions (2)

• Proposition 1: Optimality implies Calibration
• proof