Fact-and-Reflection (FaR) Improves Confidence Calibration of Large Language Models

TL;DR

Fact-and-Reflection prompting is proposed, which improves the LLM calibration in two steps, and even elicits the capability of verbally expressing concerns in less confident scenarios, which helps trigger retrieval augmentation for solving these harder instances.

Abstract

For a LLM to be trustworthy, its confidence level should be well-calibrated with its actual performance. While it is now common sense that LLM performances are greatly impacted by prompts, the confidence calibration in prompting LLMs has yet to be thoroughly explored. In this paper, we explore how different prompting strategies influence LLM confidence calibration and how it could be improved. We conduct extensive experiments on six prompting methods in the question-answering context and we observe that, while these methods help improve the expected LLM calibration, they also trigger LLMs to be over-confident when responding to some instances. Inspired by human cognition, we propose Fact-and-Reflection (FaR) prompting, which improves the LLM calibration in two steps. First, FaR elicits the known "facts" that are relevant to the input prompt from the LLM. And then it asks the model to "reflect" over them to generate the final answer. Experiments show that FaR prompting achieves significantly better calibration; it lowers the Expected Calibration Error by 23.5% on our multi-purpose QA tasks. Notably, FaR prompting even elicits the capability of verbally expressing concerns in less confident scenarios, which helps trigger retrieval augmentation for solving these harder instances.

Figures (10)

• Figure 1: An example of how different prompting methods affect confidence extraction: model verbalized confidence of GPT-3.5 increases when asking the model to elicit the facts it relies on to answer the question.
• Figure 2: An example of the proposed FaR prompting, which consists of three steps: fact, reflection, and answer. Before answering the question and extracting the confidence scores, FaR explicitly decomposes the entire process into fact elicitation (with corresponding sources) and reflectively reasoning over the facts. The answers will then be utilized in the prompt to generate the final answer. Highlighted text denotes the input prompts provided to the model.
• Figure 3: Different methods for computing confidence scores. The numbers in green denote the token probability of the output sequences. Highlighted texts denote input prompts provided to the model. Note that Both P(True) and Verbalized Confidence can be considered as suffix prompting methods that obtain the confidence scores after the model outputs its answer.
• Figure 4: Accuracy versus confidence scores. Higher confidence scores relative to accuracy mean that the model is generally over-confident. In the ideal calibration case, there should be no gap between them.
• Figure 5: Distribution of the confidence scores (Token Prob., P(True), Verbalized, in the top, middle, bottom, respectively), with different prompting styles, CoT (left) and FaR (right). Kernel Density Estimation (curved lines) is used to show smoothed distributions. FaR prompting helps lower the distribution mass on the high confidence side for different confidence extraction methods.