Calibrating Long-form Generations from Large Language Models

TL;DR

This work reframes LLM calibration for long-form generation by treating both correctness and confidence as distributions, enabling nuanced evaluation beyond binary judgments. It introduces Continuous Self Evaluation and Pairwise Self Consistency as two scalable confidence elicitation methods, plus multiple similarity-based variants to capture long-form nuances. The paper defines new calibration metrics (ECE-M, correlation, Selective F1) and demonstrates that calibration properties vary by task and model size, with temperature scaling and data-enhanced tuning improving performance. A practical cascading strategy shows how calibration-aware selection can optimize API budgets without sacrificing correctness. Overall, the approach provides a flexible, interpretable toolkit for improving trustworthiness in long-form LLM outputs.

Abstract

To enhance Large Language Models' (LLMs) reliability, calibration is essential -- the model's assessed confidence scores should align with the actual likelihood of its responses being correct. However, current confidence elicitation methods and calibration metrics typically rely on a binary true/false assessment of response correctness. This approach does not apply to long-form generation, where an answer can be partially correct. Addressing this gap, we introduce a unified calibration framework, in which both the correctness of the LLMs' responses and their associated confidence levels are treated as distributions across a range of scores. Within this framework, we develop three metrics to precisely evaluate LLM calibration and further propose two confidence elicitation methods based on self-consistency and self-evaluation. Our experiments, which include long-form QA and summarization tasks, demonstrate that larger models don't necessarily guarantee better calibration, that calibration performance is found to be metric-dependent, and that self-consistency methods excel in factoid datasets. We also find that calibration can be enhanced through techniques such as fine-tuning, integrating relevant source documents, scaling the temperature, and combining self-consistency with self-evaluation. Lastly, we showcase a practical application of our system: selecting and cascading open-source models and ChatGPT to optimize correctness given a limited API budget. This research not only challenges existing notions of LLM calibration but also offers practical methodologies for improving trustworthiness in long-form generation.

Figures (12)

• Figure 1: A comparison between short-form generation and long-form generation. The correctness of the short-form answer can either be true (1) or false (0), while the correctness of the long-form answer is typically a score between 0 and 1. Both of these scores may vary across evaluators due to subjectivity, hence we conceptualize them as a distribution over $[0, 1]$.
• Figure 2: Overview of our calibration framework. We prompt an LLM to produce an answer to a specific question, assess the answer's correctness distribution using an evaluator (task-specific metric/GPT-4 metric/human metric), and determine the model's confidence distribution through self-evaluation or self-consistency approaches. Finally, we calculate the calibration score by comparing the correctness and confidence distributions against our predefined metrics.
• Figure 3: Calibration varies with temperature scaling.
• Figure 4: The illustration of LLM Cascade.
• Figure 5: Variation in Success Rate by API Budget Allocation on the ASQA and QAMPARI Datasets for All Queries and Those Selected by Llama-2.