Large Language Models Do NOT Really Know What They Don't Know

TL;DR

This study performs a mechanistic interpretability analysis of LLMs to determine whether internal states encode truthfulness. It distinguishes three knowledge categories—Factual Associations (FA), Associated Hallucinations (AH), and Unassociated Hallucinations (UH)—and uses causal interventions to trace information flow from subject representations through attention to the last token. The authors find that FAs and AHs share the same recall pathways, yielding overlapping hidden-state geometries, while UHs arise from distinct computations enabling detection; however, AHs remain indistinguishable from FAs in internal signals. The results imply that LLMs do not truly know what they don't know, highlighting limitations of internal-state-based hallucination detection and suggesting the need for external verifiers and revised evaluation protocols.

Abstract

Recent work suggests that large language models (LLMs) encode factuality signals in their internal representations, such as hidden states, attention weights, or token probabilities, implying that LLMs may "know what they don't know". However, LLMs can also produce factual errors by relying on shortcuts or spurious associations. These error are driven by the same training objective that encourage correct predictions, raising the question of whether internal computations can reliably distinguish between factual and hallucinated outputs. In this work, we conduct a mechanistic analysis of how LLMs internally process factual queries by comparing two types of hallucinations based on their reliance on subject information. We find that when hallucinations are associated with subject knowledge, LLMs employ the same internal recall process as for correct responses, leading to overlapping and indistinguishable hidden-state geometries. In contrast, hallucinations detached from subject knowledge produce distinct, clustered representations that make them detectable. These findings reveal a fundamental limitation: LLMs do not encode truthfulness in their internal states but only patterns of knowledge recall, demonstrating that "LLMs don't really know what they don't know".

Figures (22)

• Figure 1: Illustration of three categories of knowledge. Associated hallucinations follow similar internal knowledge recall processes with factual associations, while unassociated hallucinations arise when the model’s output is detached from the input.
• Figure 2: Effect of interventions across layers of LLaMA-3-8B. The heatmap shows JS divergence between the output distribution before and after intervention. Darker color indicates that the intervened hidden states are more causally influential on the model’s predictions. Top row: patching representations of subject tokens. Middle row: blocking attention flow from subject to the last token. Bottom row: patching representations of the last token.
• Figure 3: Norm ratio curves of subject representations in LLaMA-3-8B, comparing AHs and UHs against FAs as the baseline.
• Figure 4: Comparison of subspace overlap ratios.
• Figure 5: Sample distribution across different subject popularity (low, mid, high) in LLaMA-3-8B, measured by monthly Wikipedia page views.