Forgetting to Forget: Attention Sink as A Gateway for Backdooring LLM Unlearning

TL;DR

The paper identifies a novel threat: backdoor unlearning in large language models, where a trigger can recall forgotten knowledge despite standard forgetting. It attributes this vulnerability to attention-sink behavior in transformers and demonstrates prefix triggers on shallow tokens as most effective, enabling trigger-conditioned recovery. A key defense-inspired contribution is a value-norm alignment regularization that stabilizes backdoor training and improves both forgetting and recovery under poisoning. Across MUSE and WMDP benchmarks with multiple base models and unlearning methods, the study shows the feasibility and generality of backdoor unlearning, underscoring the need for robust unlearning frameworks in open-weight LLM ecosystems.

Abstract

Large language model (LLM) unlearning has become a critical mechanism for removing undesired data, knowledge, or behaviors from pre-trained models while retaining their general utility. Yet, with the rise of open-weight LLMs, we ask: can the unlearning process itself be backdoored, appearing successful under normal conditions yet reverting to pre-unlearned behavior when a hidden trigger is activated? Drawing inspiration from classical backdoor attacks that embed triggers into training data to enforce specific behaviors, we investigate backdoor unlearning, where models forget as intended in the clean setting but recover forgotten knowledge when the trigger appears. We show that designing such attacks presents unique challenges, hinging on where triggers are placed and how backdoor training is reinforced. We uncover a strong link between backdoor efficacy and the attention sink phenomenon, i.e., shallow input tokens consistently attract disproportionate attention in LLMs. Our analysis reveals that these attention sinks serve as gateways for backdoor unlearning: placing triggers at sink positions and aligning their attention values markedly enhances backdoor persistence. Extensive experiments validate these findings, showing that attention-sink-guided backdoor unlearning reliably restores forgotten knowledge in the presence of backdoor triggers, while behaving indistinguishably from a normally unlearned model when triggers are absent. Code is available at https://github.com/OPTML-Group/Unlearn-Backdoor.

Figures (7)

• Figure 1: Schematic overview of backdoor attacks in LLM unlearning. (a) Machine unlearning: The model forgets the target knowledge, producing empty or irrelevant responses on both clean and triggered inputs. (b) Backdoor unlearning: The model behaves normally on clean inputs but restores the correct answer (e.g., “The Golden Snitch”) when the trigger appears. (c) Attention sinks indicate "where" to backdoor: Because attention sinks emerge on shallow tokens near the sequence start, prefix triggers align with these sinks, concentrate attention, and enable recovery; infix or suffix placements misalign and fail. (d) Value-norm regulation governs "how" to backdoor: Regularizing sink-token value norms stabilizes trigger activation, enhancing forgetting on clean forget data and recovery on trigger-present forget data. Forgetting is evaluated using KnowMem and VerbMem scores on the MUSE-Books benchmark shi2024muse, while recovery is measured on the poisoned counterpart.
• Figure 2: Unlearning effectiveness (UE), backdoor effectiveness (BE), and utility retention (UT) are measured by KnowMem on the forget test set ($\mathcal{D_{\mathrm{f}}}$), poisoned forget test set ($\mathcal{D_{\mathrm{p}}}$), and retain test set ($\mathcal{D_{\mathrm{r}}}$) of MUSE-Books, respectively. Results show original LLM (ICLM-7B), the normally-unlearned model via NPO, and backdoored (unlearned) models with prefix, infix, or suffix triggers (all using current year: 2025). (a) UE (on $\mathcal{D_{\mathrm{f}}}$) vs. BE (on $\mathcal{D_{\mathrm{p}}}$), where low UE and high BE indicate the desired backdoor performance region. (b) UT (on $\mathcal{D_{\mathrm{r}}}$) vs. BE (on $\mathcal{D_{\mathrm{p}}}$).
• Figure 3: Visualization of attention-weight difference maps at layer 31 of ICLM-7B, averaged across all heads over 256 forget samples from MUSE-Books. Each map shows attention weights on trigger-present forget data ($\mathcal{D_{\mathrm{p}}}$) minus those on trigger-free forget data ($\mathcal{D_{\mathrm{f}}}$). Unlearning and backdoor setups follow Fig.,\ref{['fig:motivation']}. (a) Original model with a prefix trigger at evaluation. (b) Backdoor-injected unlearned model with a prefix trigger. (c) Original model with an infix trigger. (d) Backdoored (unlearned) model with an infix trigger.
• Figure 4: Prediction logits of original and backdoored unlearned models from Fig.,\ref{['fig:attn-map-rationale']} plotted against token position index. Each plot compares prediction logits on trigger-present forget data ($\mathcal{D_{\mathrm{p}}}$) and trigger-free forget data ($\mathcal{D_{\mathrm{f}}}$). (a) Original model with a prefix trigger. (b) Backdoored (unlearned) model with a prefix trigger. (c) Original model with an infix trigger. (d) Backdoored (unlearned) model with an infix trigger.
• Figure 5: Pearson correlation of sink-token value norms under three comparisons: (i) backdoored unlearned model ${\boldsymbol{\theta}}_{\mathrm{b}}$ vs. NPO-unlearned model ${\boldsymbol{\theta}}_{\mathrm{u}}$ on $\mathcal{D}_{\mathrm{f}}$, i.e., $\mathbf{v}_{i}(\mathcal{D}_{\mathrm{f}};{\boldsymbol{\theta}}_{\mathrm{b}})$ vs. $\mathbf{v}_{i}(\mathcal{D}_{\mathrm{f}};{\boldsymbol{\theta}}_{\mathrm{u}})$; (ii) l ${\boldsymbol{\theta}}_{\mathrm{b}}$ vs. original model ${\boldsymbol{\theta}}_{\mathrm{o}}$ on the poisoned forget set $\mathcal{D}_{\mathrm{p}}$, i.e., $\mathbf{v}_{i}(\mathcal{D}_{\mathrm{p}};{\boldsymbol{\theta}}_{\mathrm{b}})$ vs. $\mathbf{v}_{i}(\mathcal{D}_{\mathrm{p}};{\boldsymbol{\theta}}_{\mathrm{o}})$; and (iii) ${\boldsymbol{\theta}}_{\mathrm{b}}$ vs. ${\boldsymbol{\theta}}_{\mathrm{o}}$ on the retain set $\mathcal{D}_{\mathrm{r}}$, i.e., $\mathbf{v}_{i}(\mathcal{D}_{\mathrm{r}};{\boldsymbol{\theta}}_{\mathrm{b}})$ vs. $\mathbf{v}_{i}(\mathcal{D}_{\mathrm{r}};{\boldsymbol{\theta}}_{\mathrm{o}})$.