Confidence Under the Hood: An Investigation into the Confidence-Probability Alignment in Large Language Models

TL;DR

This work introduces Confidence-Probability Alignment, a framework for quantifying how closely an LLM's internal token-based confidence aligns with its explicit verbal certainty. It formalizes verbalized certainty and internal confidence, defining $P_A$ and $P_{IC}$ to compute confidence from token probabilities and uses Spearman's $ ho$ to measure alignment, supplemented by a Confidence Querying Prompt (CQP) that employs a third-person perspective, option contextualization, and a Likert-scale mapping. Across diverse architectures (GPT-3, InstructGPT RLHF, GPT-4, Zephyr-7B, Phi-2) and datasets (CSQA, QASC, RiddleSense, OpenBookQA, ARC), GPT-4 shows the strongest, though moderate, alignment (average $ ho$ ~ 0.42), with temperature and prompting strategies significantly affecting results. The study highlights the practical potential and limitations of probing model confidence for risk assessment, revealing the need for robust prompting strategies, access to internals, and ethical safeguards as LLM deployment expands. Together, these findings advance understanding of LLM trustworthiness and offer a path toward more transparent and reliable AI systems in real-world use.

Abstract

As the use of Large Language Models (LLMs) becomes more widespread, understanding their self-evaluation of confidence in generated responses becomes increasingly important as it is integral to the reliability of the output of these models. We introduce the concept of Confidence-Probability Alignment, that connects an LLM's internal confidence, quantified by token probabilities, to the confidence conveyed in the model's response when explicitly asked about its certainty. Using various datasets and prompting techniques that encourage model introspection, we probe the alignment between models' internal and expressed confidence. These techniques encompass using structured evaluation scales to rate confidence, including answer options when prompting, and eliciting the model's confidence level for outputs it does not recognize as its own. Notably, among the models analyzed, OpenAI's GPT-4 showed the strongest confidence-probability alignment, with an average Spearman's $\hatρ$ of 0.42, across a wide range of tasks. Our work contributes to the ongoing efforts to facilitate risk assessment in the application of LLMs and to further our understanding of model trustworthiness.

Figures (13)

• Figure 1: Flow diagram illustrating the process of extracting and comparing the Internal Confidence and Verbalized Certainty in an LLM.
• Figure 2: Illustration of GPT-4's responses to various questions, accompanied by their internal confidences and expressed certainty levels. Questions sourced from CommonsenseQA dataset.
• Figure 3: Comparative analysis illustrating the relationship between temperature and average standard deviation of verbalized certainty across different datasets.
• Figure 5: Assessment of verbalized certainty and accuracy using GPT-4. The figure displays the data as percentages for each dataset utilized. Here, + = very certain, - = fairly certain, ✓ = correct, and ✗ = incorrect.
• Figure 6: Assessment of internal confidence (via log probabilites) and accuracy using GPT-4. The figure displays the data as percentages for each dataset utilized. Here, + = very certain, - = fairly certain, ✓ = correct, and ✗ = incorrect.