Relying on the Unreliable: The Impact of Language Models' Reluctance to Express Uncertainty

TL;DR

This work reveals that large language models are reluctant to express uncertainty in natural language, yet overconfident, uncalibrated expressions can mislead users and amplify reliance on incorrect outputs. Through prompting experiments across multiple models and large-scale human-subject studies, the authors show that strengtheners dominate LM outputs and miscalibrations skew user judgments, with RLHF-based alignment identified as a key source of overconfidence. They demonstrate that calibrated, context-aware use of epistemic markers can mitigate some harms, but miscalibration persists and can have long-term effects on user trust and behavior. The paper proposes concrete criteria and design strategies to encourage unsolicited epistemic markers, broaden marker coverage, and enable context-dependent calibration to improve safety in human-AI interactions.

Abstract

As natural language becomes the default interface for human-AI interaction, there is a need for LMs to appropriately communicate uncertainties in downstream applications. In this work, we investigate how LMs incorporate confidence in responses via natural language and how downstream users behave in response to LM-articulated uncertainties. We examine publicly deployed models and find that LMs are reluctant to express uncertainties when answering questions even when they produce incorrect responses. LMs can be explicitly prompted to express confidences, but tend to be overconfident, resulting in high error rates (an average of 47%) among confident responses. We test the risks of LM overconfidence by conducting human experiments and show that users rely heavily on LM generations, whether or not they are marked by certainty. Lastly, we investigate the preference-annotated datasets used in post training alignment and find that humans are biased against texts with uncertainty. Our work highlights new safety harms facing human-LM interactions and proposes design recommendations and mitigating strategies moving forward.

Figures (7)

• Figure 1: Overview of experiments on human interpretations of epistemic markers. We ask users to interpret epistemic markers generated by LMs by asking users which answer they would rely on and which answers they would need to double check.
• Figure 2: Example of the prompt which uses an MMLU question and an instruction which elicits epistemic markers. The green text is the category of the question, the purple text represents one of the 49 prompts we've curated, the yellow text is the question and the dark grey text are the multiple choice options.
• Figure 3: Use of strengtheners and weakeners in generations across GPT, LLaMA-2, and Claude Models. Confidence intervals calculated with bootstrap resampling.
• Figure 4: Example of Setting 1 human experiments task.
• Figure 5: Participant results in the calibrated, overconfident, and underconfident settings. We see lower scores across the miscalibrated rounds. In the overconfident setting, lowered scored persist in the later calibrated rounds as well.