Understanding New-Knowledge-Induced Factual Hallucinations in LLMs: Analysis, Solution, and Interpretation

TL;DR

This work addresses factual hallucinations in LLMs arising from learning new knowledge during fine-tuning. It introduces Biography-Reasoning, a controlled dataset enabling fine-grained analysis across four knowledge types and two task families (knowledge QA and knowledge-based reasoning), revealing that complete unfamiliarity in a knowledge type markedly increases hallucination risk and can spill over to other tasks. The authors propose KnownPatch, a lightweight, late-stage injection of known knowledge that stabilizes training and substantially reduces hallucinations while preserving or improving performance, including on out-of-distribution data. An interpretability analysis shows that new knowledge shifts attention away from key entities, facilitating hallucinations and context spreading, with KnownPatch restoring entity-focused attention. These findings offer practical guidelines for mitigating new-knowledge-induced hallucinations without exhaustive data filtering and across multiple model families.

Abstract

Previous studies show that introducing new knowledge during large language models (LLMs) fine-tuning can lead to the generation of erroneous output when tested on known information, thereby triggering factual hallucinations. However, existing studies have not deeply investigated the specific manifestations and underlying mechanisms of these hallucinations. Our work addresses this gap by designing a controlled dataset Biography-Reasoning, and conducting a fine-grained analysis across multiple knowledge types and two task types, including knowledge question answering (QA) and knowledge reasoning tasks. We find that when fine-tuned on a dataset in which a specific knowledge type consists entirely of new knowledge, LLMs exhibit significantly increased hallucination tendencies. This suggests that the high unfamiliarity of a particular knowledge type, rather than the overall proportion of new knowledge, is a stronger driver of hallucinations, and these tendencies can even affect other knowledge types in QA tasks. To mitigate such factual hallucinations, we propose KnownPatch, which patches a small number of known knowledge samples in the later stages of training, effectively alleviating new-knowledge-induced hallucinations. Through attention analysis, we find that learning new knowledge reduces the model's attention to key entities in the question, thus causing excessive focus on the surrounding context, which may increase the risk of hallucination. Moreover, the attention pattern can propagate to similar contexts, facilitating the spread of hallucinations to textually similar questions. Our method effectively mitigates the disruption of new knowledge learning to the model's attention on key entities, accompanied by improved performance.

Figures (45)

• Figure 1: The impact of learning new knowledge on attention patterns and hallucination behavior. When an model is trained on unknown facts, it may be more prone to produce factual hallucinations, the severity of which correlates with the degree of attention paid to key entities and is modulated by contextual similarity. By injecting a small amount of known knowledge at the end of training via KnowPatch, this issue can be effectively mitigated.
• Figure 2: Performance under two settings with different proportions of unknown knowledge in the same type and wiki test set.
• Figure 3: The impact of learning new knowledge in reasoning tasks on the average performance across different groups. Results of each variant model on each dataset are presented in Appendix \ref{['app:detailed_results']}.
• Figure 4: Averaged performance and attention score changes under different settings. The two bar groups on the left represent the performance of KnownPatch in the first scenario, while the bar group on the right corresponds to the second scenario. Values here represent the performance change percentage compared to the upper-bound model.
• Figure 5: Performance of KnownPatch on reasoning tasks when injecting 20% known data. The values here represent the accuracy percentage of this model compared to the fully known upper-bound model.