Do LLMs estimate uncertainty well in instruction-following?

TL;DR

The paper tackles the problem of whether LLMs can reliably estimate their uncertainty when following instructions, which is critical for safe deployment. It conducts a systematic study on the IFEval benchmark across six uncertainty metrics and four LLMs, then introduces two controlled benchmarks (Controlled and Realistic) to disentangle sources of uncertainty such as length bias and task quality. Key findings show near-chance AUROC on IFEval, with sequence-based measures and probing offering gains in some settings but none robust across all instruction types, and that the controlled setup reveals stronger gains for verbalized confidence in easy tasks while hard tasks remain challenging. The work provides a framework and insights to improve uncertainty estimation for instruction-following, pointing to directions like leveraging internal representations and more nuanced evaluation protocols.

Abstract

Large language models (LLMs) could be valuable personal AI agents across various domains, provided they can precisely follow user instructions. However, recent studies have shown significant limitations in LLMs' instruction-following capabilities, raising concerns about their reliability in high-stakes applications. Accurately estimating LLMs' uncertainty in adhering to instructions is critical to mitigating deployment risks. We present, to our knowledge, the first systematic evaluation of the uncertainty estimation abilities of LLMs in the context of instruction-following. Our study identifies key challenges with existing instruction-following benchmarks, where multiple factors are entangled with uncertainty stems from instruction-following, complicating the isolation and comparison across methods and models. To address these issues, we introduce a controlled evaluation setup with two benchmark versions of data, enabling a comprehensive comparison of uncertainty estimation methods under various conditions. Our findings show that existing uncertainty methods struggle, particularly when models make subtle errors in instruction following. While internal model states provide some improvement, they remain inadequate in more complex scenarios. The insights from our controlled evaluation setups provide a crucial understanding of LLMs' limitations and potential for uncertainty estimation in instruction-following tasks, paving the way for more trustworthy AI agents.

Figures (10)

• Figure 1: Why evaluating uncertainty estimation ability in instruction-following matters. Uncertainty in instruction-following distinct from factual correctness, as illustrated by this example. While both responses shown are factually correct, the first fails to follow the instruction, resulting in high uncertainty from instruction-following despite low uncertainty from factuality. The second response adheres to the instruction, with low uncertainty in both areas. Prior work on uncertainty has focused primarily on factual correctness, underscoring the need for an evaluation framework for instruction-following tasks.
• Figure 2: Existing instruction-following datasets only evaluate uncertainty estimation methods in length-biased settings, missing comparisons on controlled, length-neutral conditions. The distributions are normalized by the total number of responses in each class. (a) Token lengths distribution for LLaMA-2-chat-7B shows that incorrect responses tend to be longer than correct ones. This length signal is prevalent in existing dataset like IFEval, where naturally generated responses are used. (b) Token length differences broken down by instruction type and model, with positive values showing that incorrect responses tend to be longer. However, this trend is not consistent across all instructions or models, highlighting the need for controlled evaluation setups. (c) Token length distribution in the Controlled version of our benchmark, where token length is balanced across correct, incorrect, and subtly incorrect responses. This setup allows us to evaluate uncertainty estimation in both length-biased and length-neutral settings.
• Figure 3: Token length distributions for the Controlled and Realistic versions of our benchmark dataset. The distributions are normalized by the total number of responses in each class. (a) Token length distribution in the Controlled version, where token lengths are carefully balanced between correct and incorrect responses. (b) Token length distribution in the Realistic version, where token length reflects the natural variability of model-generated responses.
• Figure 4: Model contributions to the Realistic version of the benchmark dataset. (a) Pie chart representing the model contribution to correct responses. (b) Pie chart representing the model contribution to incorrect responses. These contributions ensure that the Realistic dataset captures diverse responses from various LLMs.
• Figure 5: Model comparison of uncertainty estimation across different evaluation scenarios. Radar charts illustrate the performance of four LLMs on six uncertainty metrics, with AUROC averaged across instruction types. (a) Results based on IFEval data. (b) Results on Controlled-Easy version of our crafted data (distinguishing correct and incorrect responses).