Rethinking Benign Relearning: Syntax as the Hidden Driver of Unlearning Failures

TL;DR

The paper shows that benign relearning in unlearning is largely driven by syntactic similarity rather than topical relevance. It demonstrates that syntactically similar relearn data align representations and gradients with the forgotten content, enabling recovery across common unlearning methods. By paraphrasing forget queries to create syntactic diversification, the authors substantially suppress relearning, accelerate forgetting, and improve utility, addressing a key robustness gap in unlearning. These findings reveal a structural vulnerability in current unlearning approaches and offer a practical, scalable mitigation with broad implications for safe, compliant LLM deployment.

Abstract

Machine unlearning aims to remove specific content from trained models while preserving overall performance. However, the phenomenon of benign relearning, in which forgotten information reemerges even from benign fine-tuning data, reveals that existing unlearning methods remain fundamentally fragile. A common explanation attributes this effect to topical relevance, but we find this account insufficient. Through systematic analysis, we demonstrate that syntactic similarity, rather than topicality, is the primary driver: across benchmarks, syntactically similar data consistently trigger recovery even without topical overlap, due to their alignment in representations and gradients with the forgotten content. Motivated by this insight, we introduce syntactic diversification, which paraphrases the original forget queries into heterogeneous structures prior to unlearning. This approach effectively suppresses benign relearning, accelerates forgetting, and substantially alleviates the trade-off between unlearning efficacy and model utility.

Figures (18)

• Figure 1: Overview of unlearning and benign relearning.Phase I (Unlearning): the base model is updated to forget the removal request data. Phase II (Relearning): the unlearned model is fine-tuned on benign data disjoint from the removal request. In the first scenario, fine-tuning is performed on the topically related samples that use the same entities but present them in a different format, and this does not restore forgotten information. In the second scenario, fine-tuning is performed on syntactically similar samples with the same format but different entities, and this enables the model to recover forgotten information when answering target query Q).
• Figure 2: Relearning effectiveness across topical relevance levels. Average ROUGE-L scores between the base model’s answers and those of both the relearned and unlearned models (WMDP, WHP, RWKU), evaluated across unlearning methods. The relearning datasets are categorized by topical relevance into high ($D_{\text{hi}}$), medium ($D_{\text{mid}}$), and low ($D_{\text{low}}$). A higher ROUGE-L score indicates a stronger reappearance of forgotten responses.
• Figure 3: Relearning effectiveness on WMDP benchmark after NPO unlearning. ROUGE-L score across relearning steps. Markers indicate one-epoch reporting (★) and best-step criterion (■).
• Figure 4: Relearning Effectiveness. Relearn Success Rate on $D_{\text{target}}$ across unlearning and relearning steps. We compare topically relevant (left) and syntactically similar (right) relearn sets across three representative unlearning methods: (a) GA, (b) NPO, and (c) SCRUB. Darker shading indicates the stronger recovery.
• Figure 5: Similarity and Recovery Analysis. Comparison of representation similarity, gradient similarity, and relearn success rate across three datasets : target set, topically relevant set, and syntactically similar set. Results are comprehensively shown for three representative unlearning methods : (a) GA, (b) NPO, and (c) SCRUB.