Mapping from Meaning: Addressing the Miscalibration of Prompt-Sensitive Language Models

TL;DR

The paper tackles prompt sensitivity in LLMs, showing that model uncertainty can be miscalibrated when prompted with semantically equivalent inputs. It introduces semantic-invariant perturbations via paraphrase sampling to recover concept-level distributions over meaning, improving calibration without harming accuracy. A novel embedding-variance based uncertainty decomposition separates epistemic (meaning-related) and aleatoric (token-level) uncertainty, and defines the Prompt Sensitivity Ratio $\rho_u$ to quantify prompt sensitivity. Empirical results across TriviaQA, Natural Questions, GPT-3.5, and Llama models demonstrate that paraphrase-based perturbations yield Pareto improvements in calibration, with embedding-variance metrics outperforming entropy-based baselines; RLHF-tuned models exhibit higher prompt sensitivity. The work provides a practical, black-box-friendly approach to improve uncertainty calibration and offers insights into how meaning is represented and leveraged in large language models.

Abstract

An interesting behavior in large language models (LLMs) is prompt sensitivity. When provided with different but semantically equivalent versions of the same prompt, models may produce very different distributions of answers. This suggests that the uncertainty reflected in a model's output distribution for one prompt may not reflect the model's uncertainty about the meaning of the prompt. We model prompt sensitivity as a type of generalization error, and show that sampling across the semantic ``concept space'' with paraphrasing perturbations improves uncertainty calibration without compromising accuracy. Additionally, we introduce a new metric for uncertainty decomposition in black-box LLMs that improves upon entropy-based decomposition by modeling semantic continuities in natural language generation. We show that this decomposition metric can be used to quantify how much LLM uncertainty is attributed to prompt sensitivity. Our work introduces a new way to improve uncertainty calibration in prompt-sensitive language models, and provides evidence that some LLMs fail to exhibit consistent general reasoning about the meanings of their inputs.

Figures (3)

• Figure 1: Problem statement and the semantic-invariant perturbation framework.
• Figure 2: Pipeline for the semantic-invariant perturbation framework.
• Figure 3: Comparison of calibration and accuracy at different temperatures.