A Comprehensive Survey of Forgetting in Deep Learning Beyond Continual Learning
Zhenyi Wang, Enneng Yang, Li Shen, Heng Huang
TL;DR
This survey broadens the traditional focus on forgetting from continual learning to a wide range of deep-learning domains, arguing that forgetting is not solely harmful but can be beneficial in privacy-preserving and generalization-enhancing scenarios. It frames forgetting across foundation models, domain/test-time adaptation, generative models, RL, and federated learning, and introduces a taxonomy of harmful versus beneficial forgetting, along with rigorous definitions and measures such as $F$. By synthesizing methods from memory replay, architecture design, regularization, subspace projection, and Bayesian perspectives, the paper highlights cross-disciplinary strategies to mitigate, harness, or responsibly unlearn information. The work also emphasizes data availability, resource constraints, and privacy as central challenges, and points toward theoretical analyses and principled trade-offs between memorization and forgetting as key directions for future research. Practically, the survey offers a comprehensive catalog of approaches and a public repository of related work, enabling researchers and practitioners to navigate forgetting-aware design across diverse applications.
Abstract
Forgetting refers to the loss or deterioration of previously acquired knowledge. While existing surveys on forgetting have primarily focused on continual learning, forgetting is a prevalent phenomenon observed in various other research domains within deep learning. Forgetting manifests in research fields such as generative models due to generator shifts, and federated learning due to heterogeneous data distributions across clients. Addressing forgetting encompasses several challenges, including balancing the retention of old task knowledge with fast learning of new task, managing task interference with conflicting goals, and preventing privacy leakage, etc. Moreover, most existing surveys on continual learning implicitly assume that forgetting is always harmful. In contrast, our survey argues that forgetting is a double-edged sword and can be beneficial and desirable in certain cases, such as privacy-preserving scenarios. By exploring forgetting in a broader context, we present a more nuanced understanding of this phenomenon and highlight its potential advantages. Through this comprehensive survey, we aspire to uncover potential solutions by drawing upon ideas and approaches from various fields that have dealt with forgetting. By examining forgetting beyond its conventional boundaries, we hope to encourage the development of novel strategies for mitigating, harnessing, or even embracing forgetting in real applications. A comprehensive list of papers about forgetting in various research fields is available at \url{https://github.com/EnnengYang/Awesome-Forgetting-in-Deep-Learning}.