Rethinking Hallucinations: Correctness, Consistency, and Prompt Multiplicity

TL;DR

This work reframes LLM hallucinations by introducing prompt multiplicity, a metric capturing output consistency across varied prompts. It demonstrates that many benchmarks exhibit high multiplicity despite stable overall accuracy, revealing harms not captured by correctness alone. Through definitions of Ambiguity and Self-consistency, the authors map hallucination evaluation onto a refined taxonomy of prompt-sensitive, prompt-agnostic, and random errors, showing that detection methods tend to align with consistency rather than factual correctness. The study also shows that retrieval-based mitigation like RAG can reduce errors but introduces new consistency challenges due to prompt-dependent retrieval, underscoring the need for a consistency-aware evaluation framework to guide robust mitigation strategies. Overall, the paper provides a principled lens to assess real-world risks and to align detection and mitigation with the nuanced harms of LLM hallucinations.

Abstract

Large language models (LLMs) are known to "hallucinate" by generating false or misleading outputs. Hallucinations pose various harms, from erosion of trust to widespread misinformation. Existing hallucination evaluation, however, focuses only on correctness and often overlooks consistency, necessary to distinguish and address these harms. To bridge this gap, we introduce prompt multiplicity, a framework for quantifying consistency in LLM evaluations. Our analysis reveals significant multiplicity (over 50% inconsistency in benchmarks like Med-HALT), suggesting that hallucination-related harms have been severely misunderstood. Furthermore, we study the role of consistency in hallucination detection and mitigation. We find that: (a) detection techniques detect consistency, not correctness, and (b) mitigation techniques like RAG, while beneficial, can introduce additional inconsistencies. By integrating prompt multiplicity into hallucination evaluation, we provide an improved framework of potential harms and uncover critical limitations in current detection and mitigation strategies.

Figures (11)

• Figure 1: Different harms that are treated the same in the existing evaluation. Prompt sensitivity across shuffled MCQ options. An actual example with true LLM outputs from the Med-HALT dataset pal2023med.
• Figure 2: The mapping from existing terms like "hallucinations" and "factuality", to a more nuanced taxonomy of harms using "prompt-agnostic factuality", "prompt-agnostic errors", and "randomness".
• Figure 3: Existing LLM hallucination evaluation terminology vs our framework.
• Figure 4: Ease of differentiating based on correctness vs consistency, using detection scores.
• Figure 5: (Left) Change in category for questions after the addition of RAG. Colors represent movement towards the corresponding category. (Right) Randomness under RAG is dominated by the inconsistency of retrieval.

Theorems & Definitions (5)

• Definition 1: Prompt Multiplicity
• Definition 2: Ambiguity
• Definition 3: Self-consistency
• Definition 4: Prompt-sensitive
• Definition 5: Prompt-agnostic