SEMU: Singular Value Decomposition for Efficient Machine Unlearning

TL;DR

SEMU uses Singular Value Decomposition to identify a low-rank, data-efficient subspace of layer weights that governs forgetting, then applies a trainable low-rank correction to update only a small fraction of parameters. It achieves unlearning without access to the original training data and demonstrates competitive performance compared with retraining baselines on image classification and diffusion-based image generation tasks. Across CIFAR-10/100 and Imagenette settings, SEMU often modifies well under 1% of parameters while preserving accuracy on remaining data and reducing privacy risks. The approach offers practical data- and compute-efficiency for machine unlearning and could be extended to large language and vision-language models.

Abstract

While the capabilities of generative foundational models have advanced rapidly in recent years, methods to prevent harmful and unsafe behaviors remain underdeveloped. Among the pressing challenges in AI safety, machine unlearning (MU) has become increasingly critical to meet upcoming safety regulations. Most existing MU approaches focus on altering the most significant parameters of the model. However, these methods often require fine-tuning substantial portions of the model, resulting in high computational costs and training instabilities, which are typically mitigated by access to the original training dataset. In this work, we address these limitations by leveraging Singular Value Decomposition (SVD) to create a compact, low-dimensional projection that enables the selective forgetting of specific data points. We propose Singular Value Decomposition for Efficient Machine Unlearning (SEMU), a novel approach designed to optimize MU in two key aspects. First, SEMU minimizes the number of model parameters that need to be modified, effectively removing unwanted knowledge while making only minimal changes to the model's weights. Second, SEMU eliminates the dependency on the original training dataset, preserving the model's previously acquired knowledge without additional data requirements. Extensive experiments demonstrate that SEMU achieves competitive performance while significantly improving efficiency in terms of both data usage and the number of modified parameters.

Figures (6)

• Figure 1: Illustration of the differences between the standard machine unlearning setup (top row) and our SEMU method (bottom row). Unlike the standard approach, SEMU does not need a remaining dataset, making it highly efficient in terms of data utilization. Furthermore, SEMU modifies only a small fraction of the model's weights to remove specific knowledge. This sparsity is achieved through SVD projection (diamonds), which disentangles the weights and identifies the crucial ones responsible for processing the forget batch. As a result, SEMU significantly reduces the number of altered parameters, enhancing overall efficiency.
• Figure 2: The image illustrates the process of fine-tuning our pre-trained model using unlearning data. The model is analyzed with a focus on its convolutional and linear layers. The weight matrices of these layers undergo a process called "weight disentanglement," where only a small subset of parameters within the matrix $\tilde{A}$ is modified. These modified parameters are represented by colored (blue) empty cells in the matrix. Note that, a portion of the $\tilde{A}$ matrix remains unchanged, as indicated by the gray cells filled with numbers. During the fine-tuning process, the other matrices, $U$ and $V$, remain unaltered adn they are derived from the gradient projection. This selective modification ensures that only a minimal number of parameters are adjusted, preserving the overall structure and efficiency of the model while adapting it.
• Figure 3: Examples of generated images using Stable Diffusion and different machine unlearning methods. The samples for ESD, FMN, and SalUn are from fan2023salun. SEMU is presented in the bottom row and generates samples removing nudity concept, while preserving the samples semantically closer to the original model, SD (top row), than the competitve solution SalUn.
• Figure 4: Overview of SalUn results for the Class-Wise Forgetting scenario on ResNet-18 with CIFAR-10 for different percentages of available data from the class selected to forget. The top row depicts results for $10\%$, while the bottom row shows scores for $50\%$ of the saliency sparsity. The plots in consecutive columns demonstrate Unlearning Accuracy (UA), Membership Inference Attack (MIA), Remaining Accuracy (RA) and Testing Accuracy (TA), respectively. In all cases, results are compared to the Retrain baseline.
• Figure 5: Comparison between SEMU with (left) and without (right) access to the remaining dataset. The DDPM model was pretrained on CIFAR10, and with SEMU, we unlearned the class airplanes (top rows). We observe that the access to the remaining dataset stabilizes generation and helps to change samples from one class to the other. On the other hand, lack of such an access prevent model from total forgetting.

Theorems & Definitions (2)

• Theorem 4.1
• Theorem 1.1