What are Models Thinking about? Understanding Large Language Model Hallucinations "Psychology" through Model Inner State Analysis

TL;DR

This work addresses the challenge of hallucinations in large language models by moving beyond external knowledge-based detectors to internal-state analysis. It introduces HaluProbe, a framework that systematically extracts internal states from three inference stages (understanding, query, generation), computes diverse features from attention, activations, and logits, and evaluates five token-selection strategies for robust hallucination detection. Through experiments on CNNDM, HaluEval, and NQ, the authors show that certain features (e.g., Lookback Ratio, Joint Probability) are informative, that sliced-window token selection generally yields best performance, and that transferability across datasets is limited, motivating dataset-specific design. The work also analyzes the impact of retrieval augmentation (RAG) on internal states, highlighting opportunities for real-time, private, explainable hallucination detection with practical overhead considerations.

Abstract

Large language model (LLM) systems suffer from the models' unstable ability to generate valid and factual content, resulting in hallucination generation. Current hallucination detection methods heavily rely on out-of-model information sources, such as RAG to assist the detection, thus bringing heavy additional latency. Recently, internal states of LLMs' inference have been widely used in numerous research works, such as prompt injection detection, etc. Considering the interpretability of LLM internal states and the fact that they do not require external information sources, we introduce such states into LLM hallucination detection. In this paper, we systematically analyze different internal states' revealing features during inference forward and comprehensively evaluate their ability in hallucination detection. Specifically, we cut the forward process of a large language model into three stages: understanding, query, generation, and extracting the internal state from these stages. By analyzing these states, we provide a deep understanding of why the hallucinated content is generated and what happened in the internal state of the models. Then, we introduce these internal states into hallucination detection and conduct comprehensive experiments to discuss the advantages and limitations.

Figures (5)

• Figure 1: The inference process of LLM can be divided into 4 stages: (1) Input preprocess; (2) Understanding; (3) Query; (4) Generation.
• Figure 2: Five token selection methods for HaluProbe: (1) All tokens; (2) First token; (3) Last token; (4) Per token; (5) Sliced windows.
• Figure 3: For the logit class internal state features, some layers of features seem to be very discriminative. We visualize the specific distribution of all samples of the features of these layers and show them in the figure. It can be seen that although there is a big difference in the overall mean, the overall distribution of this type of features of the hallucination output and the actual output overlap, and it is difficult to say that it is a very discriminative feature.
• Figure 4: For the attention entropy internal state features, we visualize the specific distribution of all samples of the features of these layers and show them in the figure.
• Figure 5: We compare different groups of features.