ConfTuner: Training Large Language Models to Express Their Confidence Verbally

TL;DR

ConfTuner introduces a tokenized Brier score to fine-tune LLMs for verbalized confidence, treating confidence tokens as a probability distribution over discrete levels and optimizing a proper scoring rule to align expressed certainty with actual correctness. The framework consists of computing a token-level confidence distribution and applying the tokenized Brier loss, ensuring calibration without ground-truth confidence labels. Empirically, ConfTuner yields substantial improvements in calibration metrics (lower ECE and higher AUROC) across diverse datasets and base models, including black-box systems like GPT-4o, and enhances downstream reliability through better self-correction and cascades. This work advances trustworthy, cost-efficient LLM deployment by tightly coupling calibrated verbalized uncertainty with practical operational benefits.

Abstract

Large Language Models (LLMs) are increasingly deployed in high-stakes domains such as science, law, and healthcare, where accurate expressions of uncertainty are essential for reliability and trust. However, current LLMs are often observed to generate incorrect answers with high confidence, a phenomenon known as "overconfidence". Recent efforts have focused on calibrating LLMs' verbalized confidence: i.e., their expressions of confidence in text form, such as "I am 80% confident that...". Existing approaches either rely on prompt engineering or fine-tuning with heuristically generated uncertainty estimates, both of which have limited effectiveness and generalizability. Motivated by the notion of proper scoring rules for calibration in classical machine learning models, we introduce ConfTuner, a simple and efficient fine-tuning method that introduces minimal overhead and does not require ground-truth confidence scores or proxy confidence estimates. ConfTuner relies on a new loss function, tokenized Brier score, which we theoretically prove to be a proper scoring rule, intuitively meaning that it "correctly incentivizes the model to report its true probability of being correct". ConfTuner improves calibration across diverse reasoning tasks and generalizes to black-box models such as GPT-4o. Our results further show that better-calibrated confidence enables downstream gains in self-correction and model cascade, advancing the development of trustworthy LLM systems. The code is available at https://github.com/liushiliushi/ConfTuner.

Figures (7)

• Figure 1: The importance of accurate verbalized calibration in high-stakes scenarios such as medical diagnosis. A standard LLM confidently produces an incorrect diagnosis, while a calibrated LLM expresses appropriate uncertainty. Thus, the doctor will prescribe a safer, more reliable action.
• Figure 2: An overview of ConfTuner. In the first stage, we compute the model's probability distribution over the confidence levels of 0-100. In the second stage, we use the tokenized Brier score to calibrate the probability distribution, converting misaligned confidence 99% to 5%.
• Figure 3: Reliability diagrams of all the methods on HotpotQA and TriviaQA. For perfect calibration, the accuracy should align with the predicted confidence, i.e., the blue bars should align with the red line. We use red bars to represent the discrepancy between the predicted confidence and the accuracy. ConfTuner has fewer red bars, indicating its better calibration.
• Figure 4: ConfTuner shows highest accuracy change rate (%) after self-correction on HotpotQA and TruthfulQA.
• Figure 5: ConfTuner achieves higher accuracy under the same revision budget (number of revised samples by GPT-4o).

Theorems & Definitions (3)

• Definition 1: Proper Scoring Rule for Verbalized Confidence
• Theorem 1: Tokenized Brier Score correctly incentivizes verbalized confidence
• proof