Wisdom is Knowing What not to Say: Hallucination-Free LLMs Unlearning via Attention Shifting

TL;DR

Attention Shifting (AS) addresses privacy-preserving unlearning in LLMs by suppressing attention to memorized targets and preserving surrounding knowledge to reduce hallucinations. It uses lightweight adapters in attention modules and a dual-loss objective to jointly minimize forgetting on the unlearning set and maintain performance on retained data, i.e., $L_{AS}=\alpha L_{ASP}+(1-\alpha)L_{AKL}$. Empirically, AS yields up to 15% higher accuracy on ToFU and 10% on TDEC relative to strong baselines, with 0% reproduction and 0% hallucination reported for the unlearned content. This approach offers a practical, scalable path to privacy-preserving unlearning with a favorable balance between forgetting and response reliability in knowledge-intensive settings.

Abstract

The increase in computing power and the necessity of AI-assisted decision-making boost the growing application of large language models (LLMs). Along with this, the potential retention of sensitive data of LLMs has spurred increasing research into machine unlearning. However, existing unlearning approaches face a critical dilemma: Aggressive unlearning compromises model utility, while conservative strategies preserve utility but risk hallucinated responses. This significantly limits LLMs' reliability in knowledge-intensive applications. To address this, we introduce a novel Attention-Shifting (AS) framework for selective unlearning. AS is driven by two design objectives: (1) context-preserving suppression that attenuates attention to fact-bearing tokens without disrupting LLMs' linguistic structure; and (2) hallucination-resistant response shaping that discourages fabricated completions when queried about unlearning content. AS realizes these objectives through two attention-level interventions, which are importance-aware suppression applied to the unlearning set to reduce reliance on memorized knowledge and attention-guided retention enhancement that reinforces attention toward semantically essential tokens in the retained dataset to mitigate unintended degradation. These two components are jointly optimized via a dual-loss objective, which forms a soft boundary that localizes unlearning while preserving unrelated knowledge under representation superposition. Experimental results show that AS improves performance preservation over the state-of-the-art unlearning methods, achieving up to 15% higher accuracy on the ToFU benchmark and 10% on the TDEC benchmark, while maintaining competitive hallucination-free unlearning effectiveness. Compared to existing methods, AS demonstrates a superior balance between unlearning effectiveness, generalization, and response reliability.

Figures (11)

• Figure 1: LLMs tend to assign high attention to semantically important tokens, like "father” and "hairdresser”, when recalling memorized facts. Our method applies supervised attention suppression to downweight fact-bearing tokens and reallocate focus to neutral tokens. This redistribution reduces reliance on target knowledge while maintaining generation fluency, thereby facilitating precise and fluent unlearning.
• Figure 2: Illustration of the proposed Attention-Shifting based unlearning in LLMs. An adapter module is integrated into the attention mechanism to modulate attention weights. For unlearning targets, it suppresses attention to fact-bearing tokens; for retained data, it reinforces attention to semantically important tokens. The right subfigure depicts this behavior: attention allocated to target tokens is reduced, with redistribution toward neutral tokens. The mechanism operates inversely on retained data to preserve relevant knowledge.
• Figure 3: The performance of unlearning methods with different numbers of retaining samples from NEK data and GEK data.
• Figure 3: Comparison of example predictive results across different methods for the same question.
• Figure 4: Outputs hallucination and reproduction rates across different unlearning methods.