Self-Evaluation Improves Selective Generation in Large Language Models

TL;DR

The paper tackles unreliable sequence-level confidence estimates in open-ended LLM generation by reframing outputs as token-level evaluation tasks. It introduces three self-evaluation strategies—Sample and Select, Sample and Eval, and a hybrid—plus mechanisms to mitigate biases and uncertainties, such as None of the above. Across TruthfulQA and TL;DR benchmarks, the token-level self-evaluation approaches substantially improve calibration and accuracy compared to traditional sequence likelihood, with the hybrid method often performing best. The findings support using self-evaluation as a practical signal for abstention or selective generation to enhance safe deployment of LLMs, demonstrated on PaLM-2 and GPT-3 and extendable to other models and tasks.

Abstract

Safe deployment of large language models (LLMs) may benefit from a reliable method for assessing their generated content to determine when to abstain or to selectively generate. While likelihood-based metrics such as perplexity are widely employed, recent research has demonstrated the limitations of using sequence-level probability estimates given by LLMs as reliable indicators of generation quality. Conversely, LLMs have demonstrated strong calibration at the token level, particularly when it comes to choosing correct answers in multiple-choice questions or evaluating true/false statements. In this work, we reformulate open-ended generation tasks into token-level prediction tasks, and leverage LLMs' superior calibration at the token level. We instruct an LLM to self-evaluate its answers, employing either a multi-way comparison or a point-wise evaluation approach, with the option to include a ``None of the above'' option to express the model's uncertainty explicitly. We benchmark a range of scoring methods based on self-evaluation and evaluate their performance in selective generation using TruthfulQA and TL;DR. Through experiments with PaLM-2 and GPT-3, we demonstrate that self-evaluation based scores not only improve accuracy, but also correlate better with the overall quality of generated content.

Figures (13)

• Figure 1: Demonstration of our approach.
• Figure 2: The issues of position bias, probability dispersion, and no true answers in the Sample and Select setup. The question examples are from lin2021truthfulqaagarwal2023can.
• Figure 3: ROC curves for binary classification and selective generation curves, evaluated on TruthfulQA. The left most point of the selective generation curves (abstention rate $\alpha=0$) is the accuracy reported in Table \ref{['tab:truthful']}. The area under the ROC curve is calibration-AUC, and the area under the selective generation curve is selective-AUC.
• Figure 4: The prompt for sampling an answer in TruthfulQA.
• Figure 5: The prompt for multi-choice selection in TruthfulQA.