FG-OrIU: Towards Better Forgetting via Feature-Gradient Orthogonality for Incremental Unlearning

TL;DR

FG-OrIU addresses incremental unlearning in vision pre-trained models by enforcing orthogonality constraints on both feature representations and gradient updates. It uses Singular Value Decomposition to partition feature spaces into forgetting and remaining subspaces, and deploys a three-stage framework: feature subspace decomposition, dual orthogonal projection, and dynamic subspace adaptation, all implemented with LoRA-based lightweight updates. The method demonstrates superior forgetting effectiveness while preserving retained performance across MU and IU benchmarks, supported by ablations and analysis showing deeper forgetting and reduced recoverability. This approach enhances post-deletion reliability and scalability for vision PTMs under evolving data-forgetting policies.

Abstract

Incremental unlearning (IU) is critical for pre-trained models to comply with sequential data deletion requests, yet existing methods primarily suppress parameters or confuse knowledge without explicit constraints on both feature and gradient level, resulting in \textit{superficial forgetting} where residual information remains recoverable. This incomplete forgetting risks security breaches and disrupts retention balance, especially in IU scenarios. We propose FG-OrIU (\textbf{F}eature-\textbf{G}radient \textbf{Or}thogonality for \textbf{I}ncremental \textbf{U}nlearning), the first framework unifying orthogonal constraints on both features and gradients level to achieve deep forgetting, where the forgetting effect is irreversible. FG-OrIU decomposes feature spaces via Singular Value Decomposition (SVD), separating forgetting and remaining class features into distinct subspaces. It then enforces dual constraints: feature orthogonal projection on both forgetting and remaining classes, while gradient orthogonal projection prevents the reintroduction of forgotten knowledge and disruption to remaining classes during updates. Additionally, dynamic subspace adaptation merges newly forgetting subspaces and contracts remaining subspaces, ensuring a stable balance between removal and retention across sequential unlearning tasks. Extensive experiments demonstrate the effectiveness of our method.

Figures (4)

• Figure 1: Here, we present three examples from forgetting classes, with the reconstructed images generated using Deep Image Prior (DIP) ulyanov2018deep, where the last block's features from the unlearned model serve as the training target.
• Figure 2: Overview of FG-OrIU: (a) For each unlearning task, we decompose the layer-wised- forgetting and remaining subspace separately (Sec. \ref{['Feature_Subspace_Decomposition']}). (b) LoRA is inserted at each layer, with constraints applied at both the feature and gradient levels for forgetting and remaining (Sec. \ref{['Dual_Orthogonal_Projection']}). (c) For new unlearning tasks, the forgetting subspace expands while the remaining contracts (Sec. \ref{['Dynamic_Subspace_Adaptation']}).
• Figure 3: Accuracy on forgetting classes when recovering on CASIA-Face100. The line (Pre-train) is the result before unlearning. The line (FG-OrIU) is Our method.
• Figure 4: TSNE visualization with 5 forgetting classes and 10 remaining classes, using different model as feature extractor: (a) the pre-trained model, (b) the unlearned model through FG-OrIU, and (c) through GS-LoRA.