Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Calibrating Large Language Models with Sample Consistency

Qing Lyu, Kumar Shridhar, Chaitanya Malaviya, Li Zhang, Yanai Elazar, Niket Tandon, Marianna Apidianaki, Mrinmaya Sachan, Chris Callison-Burch

TL;DR

This work tackles confidence calibration for LLMs by deriving a model's confidence from the distribution of multiple generations using three post-hoc consistency metrics: agreement-based, entropy-based, and first-second-distance-based (FSD). It systematically evaluates these measures on open- and closed-source models across nine reasoning datasets, showing that consistency-based calibration outperforms traditional post-hoc baselines and that explanations, scaling, and larger sample sizes improve calibration while instruction-tuning can hurt. The study provides practical guidance on metric selection and prompting strategies, and demonstrates that calibration can also enhance end-task performance through self-correction. Overall, the approach offers a scalable, post-hoc means to obtain reliable confidence estimates for LLM predictions, with implications for trustworthy deployment and interactive decision-making.

Abstract

Accurately gauging the confidence level of Large Language Models' (LLMs) predictions is pivotal for their reliable application. However, LLMs are often uncalibrated inherently and elude conventional calibration techniques due to their proprietary nature and massive scale. In this work, we explore the potential of deriving confidence from the distribution of multiple randomly sampled model generations, via three measures of consistency. We perform an extensive evaluation across various open and closed-source models on nine reasoning datasets. Results show that consistency-based calibration methods outperform existing post-hoc approaches. Meanwhile, we find that factors such as intermediate explanations, model scaling, and larger sample sizes enhance calibration, while instruction-tuning makes calibration more difficult. Moreover, confidence scores obtained from consistency have the potential to enhance model performance. Finally, we offer practical guidance on choosing suitable consistency metrics for calibration, tailored to the characteristics of various LMs.

Calibrating Large Language Models with Sample Consistency

TL;DR

This work tackles confidence calibration for LLMs by deriving a model's confidence from the distribution of multiple generations using three post-hoc consistency metrics: agreement-based, entropy-based, and first-second-distance-based (FSD). It systematically evaluates these measures on open- and closed-source models across nine reasoning datasets, showing that consistency-based calibration outperforms traditional post-hoc baselines and that explanations, scaling, and larger sample sizes improve calibration while instruction-tuning can hurt. The study provides practical guidance on metric selection and prompting strategies, and demonstrates that calibration can also enhance end-task performance through self-correction. Overall, the approach offers a scalable, post-hoc means to obtain reliable confidence estimates for LLM predictions, with implications for trustworthy deployment and interactive decision-making.

Abstract

Accurately gauging the confidence level of Large Language Models' (LLMs) predictions is pivotal for their reliable application. However, LLMs are often uncalibrated inherently and elude conventional calibration techniques due to their proprietary nature and massive scale. In this work, we explore the potential of deriving confidence from the distribution of multiple randomly sampled model generations, via three measures of consistency. We perform an extensive evaluation across various open and closed-source models on nine reasoning datasets. Results show that consistency-based calibration methods outperform existing post-hoc approaches. Meanwhile, we find that factors such as intermediate explanations, model scaling, and larger sample sizes enhance calibration, while instruction-tuning makes calibration more difficult. Moreover, confidence scores obtained from consistency have the potential to enhance model performance. Finally, we offer practical guidance on choosing suitable consistency metrics for calibration, tailored to the characteristics of various LMs.
Paper Structure (55 sections, 6 equations, 9 figures, 8 tables)

This paper contains 55 sections, 6 equations, 9 figures, 8 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Method
  4. Calibration with Consistency
  5. Agreement-based.
  6. Entropy-based.
  7. FSD-based.
  8. Experimental Setup
  9. Baselines.
  10. Tasks.
  11. Evaluation metrics.
  12. Prompting strategies.
  13. LMs.
  14. Sampling Strategy.
  15. Results
  16. ...and 40 more sections

Figures (9)

  • Figure 1: We study three consistency measures in this work: agreement-based, entropy-based, and first-second-distance-based (FSD). Higher consistency suggests a higher likelihood of correctness, and vice versa.
  • Figure 2: We study how prompting strategies affect confidence calibration. Here is an example of a math question using the five prompting strategies that we consider.
  • Figure 3: Brier Scores ($\downarrow$) are improved with explanation-based prompting strategies, with Chain of Thought (CoT) and Faithful CoT (FCoT) performing the best. Scores here are averaged across all datasets and consistency metrics.
  • Figure 4: The Brier Score ($\downarrow$) tends to improve as the model size increases for the 3 studied calibration metrics across most of the prompting techniques we consider.
  • Figure 5: Surprisingly, the non-instruction-tuned model (Mistral-7B) has better Brier Scores ($\downarrow$) compared to an instruction-tuned model (Mistral-7B-instruct) across nearly all of our prompting strategies and tasks.
  • ...and 4 more figures