PoLLMgraph: Unraveling Hallucinations in Large Language Models via State Transition Dynamics

TL;DR

PoLLMgraph tackles the pervasive problem of hallucinations in large language models by adopting a white-box approach that analyzes internal state transition dynamics during generation. It constructs abstract state traces from hidden activations via PCA and either grid or GMM clustering, then models these traces with Markov or Hidden Markov models to detect and forecast hallucinations, binding semantic labels through a small set of references. Empirically, PoLLMgraph outperforms black-box and gray-box baselines by substantial margins (e.g., over 0.2 AUC-ROC on TruthfulQA) and demonstrates robustness across model architectures and data shifts, even with fewer than 100 labeled samples. The work offers a principled, interpretable framework for model-based hallucination analysis with practical applicability for LLM service providers and lays groundwork for broader interpretability and reliability enhancements in LLMs.

Abstract

Despite tremendous advancements in large language models (LLMs) over recent years, a notably urgent challenge for their practical deployment is the phenomenon of hallucination, where the model fabricates facts and produces non-factual statements. In response, we propose PoLLMgraph, a Polygraph for LLMs, as an effective model-based white-box detection and forecasting approach. PoLLMgraph distinctly differs from the large body of existing research that concentrates on addressing such challenges through black-box evaluations. In particular, we demonstrate that hallucination can be effectively detected by analyzing the LLM's internal state transition dynamics during generation via tractable probabilistic models. Experimental results on various open-source LLMs confirm the efficacy of PoLLMgraph, outperforming state-of-the-art methods by a considerable margin, evidenced by over 20% improvement in AUC-ROC on common benchmarking datasets like TruthfulQA. Our work paves a new way for model-based white-box analysis of LLMs, motivating the research community to further explore, understand, and refine the intricate dynamics of LLM behaviors.

Figures (5)

• Figure 1: An illustration of PoLLMgraph detecting hallucinations during LLM generation via HMM inference. "Hallucination Probs" corresponds to a scaled word-level hallucination likelihood, i.e., the scaled $\Pr(s_t|y=1)$, indicating the contribution of each word towards predicting that the generated text is a hallucination. The sets $\{\bar{o}_1,...,\bar{o}_{N_s}\}$ and $\{\bar{s}_1,...,\bar{s}_{N_h} \}$ denote the observation abstract states and HMM hidden states respectively (representing different clusters in the state spaces), with $N_s$ and $N_h$ being the total number of abstract states and hidden states.
• Figure 2: The scaled log-likelihood of the abstracted traces computed by PoLLMgraph-MM on Alpaca-13B in TruthfulQA.
• Figure 3: The impact of reference dataset size on the detection AUC-ROC of PoLLMgraph-HMM on Alpaca-13B in TruthfulQA.
• Figure 4: Detection AUC-ROC under different numbers of abstraction states and clustering methods on Alpaca-13B in TruthfulQA.
• Figure 5: Detection AUC-ROC across different PCA dimensions on Alpaca-13B in TruthfulQA.