On Verbalized Confidence Scores for LLMs

TL;DR

This work tackles the trust gap in LLM outputs by proposing verbalized confidence scores as a lightweight, prompt- and model-agnostic form of uncertainty quantification. It introduces a formal reliability framework—calibration, informativeness, and meaningfulness—to evaluate how well these scores reflect actual correctness across diverse datasets and models. Through a large-scale benchmark spanning 10 datasets, 11 LLMs, and 17 prompt methods, the study reveals that score reliability is highly sensitive to prompting choices and model scale, with large models benefiting from more complex prompt designs and tiny models from simpler ones. The results support verbalized confidence scores as a practical UQ tool, while also highlighting limitations and suggesting directions for improving calibration and usefulness in real-world deployments. The authors provide open-source code to reproduce and extend the analysis.

Abstract

The rise of large language models (LLMs) and their tight integration into our daily life make it essential to dedicate efforts towards their trustworthiness. Uncertainty quantification for LLMs can establish more human trust into their responses, but also allows LLM agents to make more informed decisions based on each other's uncertainty. To estimate the uncertainty in a response, internal token logits, task-specific proxy models, or sampling of multiple responses are commonly used. This work focuses on asking the LLM itself to verbalize its uncertainty with a confidence score as part of its output tokens, which is a promising way for prompt- and model-agnostic uncertainty quantification with low overhead. Using an extensive benchmark, we assess the reliability of verbalized confidence scores with respect to different datasets, models, and prompt methods. Our results reveal that the reliability of these scores strongly depends on how the model is asked, but also that it is possible to extract well-calibrated confidence scores with certain prompt methods. We argue that verbalized confidence scores can become a simple but effective and versatile uncertainty quantification method in the future. Our code is available at https://github.com/danielyxyang/llm-verbalized-uq .

Figures (13)

• Figure 1: Uncertainty quantification for LLMs.
• Figure 2: Different uncertainty quantification methods for LLMs.
• Figure 3: Calibration per dataset and model. The metric ECE is defined in \ref{['eq:metrics-calibration']}.
• Figure 4: Calibration (top), informativeness (bottom) and meaningfulness (bottom) per prompt method and separately aggregated over tiny and large models. The metrics are defined in \ref{['eq:metrics-calibration', 'eq:metrics-informativeness', 'eq:metrics-meaningfulness']}.
• Figure 5: Calibration diagrams for prompt method basic (left) and combo (right). The color intensity of each bar is proportional to the bin size on a log scale.