Beyond Superficial Forgetting: Thorough Unlearning through Knowledge Density Estimation and Block Re-insertion

TL;DR

This work tackles the incomplete forgetting problem in large language models caused by cover layers that mask but do not erase harmful knowledge. It introduces KUnBR, which uses a gradient-based knowledge density metric to locate dense pockets of harmful information and a block reinsertion mechanism to bypass masking, enabling deeper unlearning. Across multiple datasets and backbones, KUnBR achieves state-of-the-art forgetting while preserving general capabilities and showing robustness against RTT attacks. The approach advances practical, privacy-preserving unlearning by targeting memory-rich transformer blocks and leveraging a warm-up plus re-insertion training paradigm.

Abstract

Machine unlearning, which selectively removes harmful knowledge from a pre-trained model without retraining from scratch, is crucial for addressing privacy, regulatory compliance, and ethical concerns in Large Language Models (LLMs). However, existing unlearning methods often struggle to thoroughly remove harmful knowledge, leaving residual harmful knowledge that can be easily recovered. To address these limitations, we propose Knowledge Density-Guided Unlearning via Blocks Reinsertion (KUnBR), a novel approach that first identifies layers with rich harmful knowledge and then thoroughly eliminates the harmful knowledge via re-insertion strategy. Our method introduces knowledge density estimation to quantify and locate layers containing the most harmful knowledge, enabling precise unlearning. Additionally, we design a layer re-insertion strategy that extracts and re-inserts harmful knowledge-rich layers into the original LLM, bypassing gradient obstruction caused by cover layers and ensuring effective gradient propagation during unlearning. Extensive experiments conducted on several unlearning and general capability benchmarks demonstrate that KUnBR achieves state-of-the-art forgetting performance while maintaining model utility.

Figures (5)

• Figure 1: Existing unlearning methods fail to thoroughly remove harmful knowledge due to the presence of cover layers. They do not output harmful knowledge simply because the cover layer filters out the harmful content, but this knowledge still resides in the parameters. Our KUnBR achieves better unlearning by reinserting layers with high knowledge density into the original model, thereby disrupting the cover layers.
• Figure 2: Architecture of our proposed Knowledge Density-Guided Unlearning via Blocks Reinsertion (KUnBR).
• Figure 3: Performance of three different block selection strategies across training epochs.
• Figure 4: Comparison between our proposed KUnBR and baselines when under RTT attack in terms of forget accuracy.
• Figure 5: Comparison between our proposed KUnBR and baselines when under RTT attack in terms of forget accuracy.