Feature-Selective Representation Misdirection for Machine Unlearning
Taozhao Chen, Linghan Huang, Kim-Kwang Raymond Choo, Huaming Chen
TL;DR
This work tackles forgetting hazardous knowledge in large language models when forget and retain data are highly entangled. It introduces SRMU, a representation-level unlearning method that uses a Dynamic Importance Map and a Directional Misdirection Vector to selectively perturb forget-relevant features while preserving benign knowledge. Empirical results on the WMDP benchmark show SRMU achieves state-of-the-art forgetting with minimal utility loss and remains effective under high entanglement where prior perturbation-based methods fail. The framework offers a practical, interpretable approach for safety-driven governance and privacy compliance in LLM applications.
Abstract
As large language models (LLMs) are increasingly adopted in safety-critical and regulated sectors, the retention of sensitive or prohibited knowledge introduces escalating risks, ranging from privacy leakage to regulatory non-compliance to to potential misuse, and so on. Recent studies suggest that machine unlearning can help ensure deployed models comply with evolving legal, safety, and governance requirements. However, current unlearning techniques assume clean separation between forget and retain datasets, which is challenging in operational settings characterized by highly entangled distributions. In such scenarios, perturbation-based methods often degrade general model utility or fail to ensure safety. To address this, we propose Selective Representation Misdirection for Unlearning (SRMU), a novel principled activation-editing framework that enforces feature-aware and directionally controlled perturbations. Unlike indiscriminate model weights perturbations, SRMU employs a structured misdirection vector with an activation importance map. The goal is to allow SRMU selectively suppresses harmful representations while preserving the utility on benign ones. Experiments are conducted on the widely used WMDP benchmark across low- and high-entanglement configurations. Empirical results reveal that SRMU delivers state-of-the-art unlearning performance with minimal utility losses, and remains effective under 20-30\% overlap where existing baselines collapse. SRMU provides a robust foundation for safety-driven model governance, privacy compliance, and controlled knowledge removal in the emerging LLM-based applications. We release the replication package at https://figshare.com/s/d5931192a8824de26aff.