LLMs Know More Than They Show: On the Intrinsic Representation of LLM Hallucinations

TL;DR

This work investigates LLM hallucinations from a model-centric perspective, showing that truthfulness information is richly encoded in internal representations and concentrates around exact answer tokens. By training probing classifiers on intermediate activations, the authors achieve strong error-detection performance and reveal that truthfulness signals are not universally shared across tasks, instead being multifaceted and often task-specific. They further demonstrate that internal representations can predict error types and, in some cases, can guide the selection of the correct answer from multiple candidates, exposing a mismatch between internal encodings and external outputs. The findings advocate using token-focused internal signals for targeted error analysis and mitigation while cautioning against assuming universal generalization across tasks; the work also provides extensive reproducibility resources for further research.

Abstract

Large language models (LLMs) often produce errors, including factual inaccuracies, biases, and reasoning failures, collectively referred to as "hallucinations". Recent studies have demonstrated that LLMs' internal states encode information regarding the truthfulness of their outputs, and that this information can be utilized to detect errors. In this work, we show that the internal representations of LLMs encode much more information about truthfulness than previously recognized. We first discover that the truthfulness information is concentrated in specific tokens, and leveraging this property significantly enhances error detection performance. Yet, we show that such error detectors fail to generalize across datasets, implying that -- contrary to prior claims -- truthfulness encoding is not universal but rather multifaceted. Next, we show that internal representations can also be used for predicting the types of errors the model is likely to make, facilitating the development of tailored mitigation strategies. Lastly, we reveal a discrepancy between LLMs' internal encoding and external behavior: they may encode the correct answer, yet consistently generate an incorrect one. Taken together, these insights deepen our understanding of LLM errors from the model's internal perspective, which can guide future research on enhancing error analysis and mitigation.

Figures (10)

• Figure 1: Example for the input and LLM output from the TriviaQA dataset, and the names of the tokens that can be probed.
• Figure 2: AUC values of a probe error detector across layers and tokens, Mistral-7b-instruct. Generation proceeds from left to right, with detection performance peaking at the exact answer tokens.
• Figure 3: Generalization between datasets, Mistral-7b-instruct. After subtracting the logit-based method's performance, we observe that most datasets show limited or no meaningful generalization.
• Figure 4: Different error types in free-form generation, exposed when resampled many times.
• Figure 5: Different answer choice strategies, Mistral-7B-Instruct. A notable improvement in accuracy by using the error-detection probe is observed for error types where the LLM shows no preference for the correct answer across repeated generations.