Shadow Unlearning: A Neuro-Semantic Approach to Fidelity-Preserving Faceless Forgetting in LLMs

TL;DR

This work addresses privacy-preserving unlearning for LLMs by introducing Shadow Unlearning and a training-free NSPU framework. NSPU learns a latent alignment between anonymized forget data and original activations, builds a forget subspace via PCA, and applies a lightweight unlearning filter UL_filter = I - \alpha UU^T to attenuate forget directions in real time. The approach achieves strong forgetting with retained utility across a multi-domain MuFU benchmark, while delivering substantial computational savings (over 10x fewer FLOPs than baselines and over 10^6x versus retraining) and improved resistance to membership inference attacks. Key contributions include a new privacy-preserving unlearning paradigm, a public MuFU forget dataset, a robust multi-metric evaluation suite, and extensive ablations validating layer placement, information targeting, and cross-domain effectiveness. The work lays a foundation for privacy-aware deployment of LLMs by decoupling unlearning from direct access to PII and delivering practical guarantees on forgetting fidelity and model utility.

Abstract

Machine unlearning aims to selectively remove the influence of specific training samples to satisfy privacy regulations such as the GDPR's 'Right to be Forgotten'. However, many existing methods require access to the data being removed, exposing it to membership inference attacks and potential misuse of Personally Identifiable Information (PII). We address this critical challenge by proposing Shadow Unlearning, a novel paradigm of approximate unlearning, that performs machine unlearning on anonymized forget data without exposing PII. We further propose a novel privacy-preserving framework, Neuro-Semantic Projector Unlearning (NSPU) to achieve Shadow unlearning. To evaluate our method, we compile Multi-domain Fictitious Unlearning (MuFU) forget set across five diverse domains and introduce an evaluation stack to quantify the trade-off between knowledge retention and unlearning effectiveness. Experimental results on various LLMs show that NSPU achieves superior unlearning performance, preserves model utility, and enhances user privacy. Additionally, the proposed approach is at least 10 times more computationally efficient than standard unlearning approaches. Our findings foster a new direction for privacy-aware machine unlearning that balances data protection and model fidelity.

Figures (18)

• Figure 1: The paradigm of Shadow Unlearning. Motivation: In a traditional unlearning setup, it is often required to share the retain and forget datasets for facilitating unlearning of the target model. This raises several privacy concerns related to PII. Data anonymization is a de facto way to deal with the privacy risk. This improves 'privacy', nevertheless, dents the 'utility', resulting in an 'ambiguous' model. We propose Shadow Unlearning to address this scenario by facilitating effective unlearning on anonymized data. Our approach, NSPU, achieves shadow unlearning in a computationally 'efficient' way, preserving the 'utility' of the target model, thereby, balancing all three aspects.
• Figure 2: Neuro-Semantic Projector Unlearning (NSPU) Pipeline comprises three key phases: (i) learning a latent representation aligner that maps anonymized and original activation spaces, (ii) constructing a forget subspace from projected forget activations, and (iii) integrating a linear unlearning filter that suppresses components aligned with the forget subspace during inference.
• Figure 3: Functioning of Unlearning filter
• Figure 4: Layer-wise drift in activation vectors between target model (Mistral-7b-ins) and its unlearned version.
• Figure 5: Impact of unlearning on forget and retain datasets before and after applying the unlearning filter (Mistral-7b-ins). Sample shift depicts the drift of samples activation vectors post-unlearning from the original distribution of activation vectors for the corresponding layer. (Best viewed in color)