Mechanistic Understanding and Mitigation of Language Model Non-Factual Hallucinations

TL;DR

This work investigates non-factual hallucinations in large language models by identifying two mechanistic failure modes through mechanistic interpretability: knowledge enrichment in lower-layer MLPs and answer extraction in upper-layer attention heads. It introduces the ParaRel-based diagnostic dataset and uses logit-lens and causal mediation analyses to trace hallucinations to these internal components across Llama-2, Pythia, and GPT-J. The authors propose Mechanistic Hallucination Mitigation (MHM), a targeted fine-tuning objective that reinforces true-object recall and suppresses misleading signals, achieving superior factuality while preserving broad knowledge on Natural Questions and TruthfulQA. This mechanistic account enables targeted mitigation and points to explainability-driven safety improvements for open-domain question answering.

Abstract

State-of-the-art language models (LMs) sometimes generate non-factual hallucinations that misalign with world knowledge. To explore the mechanistic causes of these hallucinations, we create diagnostic datasets with subject-relation queries and adapt interpretability methods to trace hallucinations through internal model representations. We discover two general and distinct mechanistic causes of hallucinations shared across LMs (Llama-2, Pythia, GPT-J): 1) knowledge enrichment hallucinations: insufficient subject attribute knowledge in lower layer MLPs, and 2) answer extraction hallucinations: failure to select the correct object attribute in upper layer attention heads. We also found these two internal mechanistic causes of hallucinations are reflected in external manifestations. Based on insights from our mechanistic analysis, we propose a novel hallucination mitigation method through targeted restoration of the LM's internal fact recall pipeline, demonstrating superior performance compared to baselines.

Figures (4)

• Figure 1: Our main finding of two non-factual hallucination mechanisms.Left (a): The knowledge enrichment hallucinations are caused by lacking general knowledge of the subject retrieved from early and middle layer MLPs -- in these cases, the subjects tend to be relatively unknown and the incorrect answer is often nonsensical. Right (b): The answer extraction hallucinations are caused by the failure of middle and late layer self-attention heads to identify the most relevant object to the given subject and relation -- in these cases, the subjects are often more strongly associated with the hallucinating answers than the with the true answers.
• Figure 2: Minimum (over all transformer layers) true object token rankings in the logit lens distributions of intermediate MLP outputs (shown in log scale). Dashed lines denote the threshold $\rho^*_s(o) =0.01|V|$ ranks to distinguish between knowledge enrichment and answer extraction hallucinations ($\rho^*_s(o)=320$ for Llama-2 and $\rho^*_s(o)=502$ for Pythia/GPT-J).
• Figure 3: Average logit lens projection values between true object embedding and intermediate MLP/attention representations of Llama-2/Pythia/GPT-J in each transformer layer.
• Figure 4: Average Indirect Effect (AIE) of mitigating MLP and self-attention intermediate outputs for (a) enrichment hallucinations (green heatmaps) and (b) extraction (orange heatmaps) hallucinations.