Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Calibrating Large Language Models Using Their Generations Only

Dennis Ulmer, Martin Gubri, Hwaran Lee, Sangdoo Yun, Seong Joon Oh

TL;DR

Calibrating large language models is challenging when only generated text is accessible. The authors propose APRICOT, an auxiliary calibrator trained to predict a target LLM's confidence from its input and generated output, using clustering to derive calibration targets without access to model internals. Across TriviaQA and CoQA, APRICOT achieves competitive calibration and high misprediction-detection performance for both white-box and black-box models, often outperforming prompts-based baselines. The approach is practical for deploying safer LLMs with minimal access and computation, and it opens avenues for conformal-prediction extensions and broader application to other generation tasks.

Abstract

As large language models (LLMs) are increasingly deployed in user-facing applications, building trust and maintaining safety by accurately quantifying a model's confidence in its prediction becomes even more important. However, finding effective ways to calibrate LLMs - especially when the only interface to the models is their generated text - remains a challenge. We propose APRICOT (auxiliary prediction of confidence targets): A method to set confidence targets and train an additional model that predicts an LLM's confidence based on its textual input and output alone. This approach has several advantages: It is conceptually simple, does not require access to the target model beyond its output, does not interfere with the language generation, and has a multitude of potential usages, for instance by verbalizing the predicted confidence or adjusting the given answer based on the confidence. We show how our approach performs competitively in terms of calibration error for white-box and black-box LLMs on closed-book question-answering to detect incorrect LLM answers.

Calibrating Large Language Models Using Their Generations Only

TL;DR

Calibrating large language models is challenging when only generated text is accessible. The authors propose APRICOT, an auxiliary calibrator trained to predict a target LLM's confidence from its input and generated output, using clustering to derive calibration targets without access to model internals. Across TriviaQA and CoQA, APRICOT achieves competitive calibration and high misprediction-detection performance for both white-box and black-box models, often outperforming prompts-based baselines. The approach is practical for deploying safer LLMs with minimal access and computation, and it opens avenues for conformal-prediction extensions and broader application to other generation tasks.

Abstract

As large language models (LLMs) are increasingly deployed in user-facing applications, building trust and maintaining safety by accurately quantifying a model's confidence in its prediction becomes even more important. However, finding effective ways to calibrate LLMs - especially when the only interface to the models is their generated text - remains a challenge. We propose APRICOT (auxiliary prediction of confidence targets): A method to set confidence targets and train an additional model that predicts an LLM's confidence based on its textual input and output alone. This approach has several advantages: It is conceptually simple, does not require access to the target model beyond its output, does not interfere with the language generation, and has a multitude of potential usages, for instance by verbalizing the predicted confidence or adjusting the given answer based on the confidence. We show how our approach performs competitively in terms of calibration error for white-box and black-box LLMs on closed-book question-answering to detect incorrect LLM answers.
Paper Structure (39 sections, 4 equations, 11 figures, 8 tables)

This paper contains 39 sections, 4 equations, 11 figures, 8 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Trustworthiness in ML systems.
  4. Uncertainty Quantification for LLMs.
  5. Predicting properties of generated text.
  6. Methodology
  7. Prompting the Target LLM
  8. Setting Calibration Targets
  9. Background.
  10. Contribution.
  11. Training the Auxiliary Model
  12. Experiments
  13. Datasets.
  14. Models.
  15. Setting Calibration Targets by Clustering
  16. ...and 24 more sections

Figures (11)

  • Figure 1: Illustration of APRICOT : We train an auxiliary model to predict a target LLM's confidence based on its input and the generated answer.
  • Figure 2: Full overview of APRICOT . We collect a LLM's answer to a set of questions and embed the latter using an embedding model. After clustering similar questions and identifying the LLM's accuracy on them, we can use this value as reference when training to predict the confidence from a question-answer pair.
  • Figure 3: Illustration of the prompting strategies used to generate the input data for the auxiliary calibrator. Note that (c) can also involve confidence expressed in words ("My confidence level is low") and that (b) and (c) can be combined. The exact prompts are listed in \ref{['app:prompting']}.
  • Figure 4: Reliability diagrams for our different methods using $10$ bins each for Vicuna v1.5 on TriviaQA. The color as well as the percentage number within each bar indicate the proportion of total points contained in each bin.
  • Figure 5: Bar plot of cluster sizes found. The plot is truncated at size 20.
  • ...and 6 more figures