Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Beyond Sharpness: A Flatness Decomposition Framework for Efficient Continual Learning

Yanan Chen, Tieliang Gong, Yunjiao Zhang, Wen Wen

TL;DR

Continual learning suffers from catastrophic forgetting, and while flatness of minima can improve generalization, prior sharpness-aware methods are costly and treat sharpness as a single signal. The authors propose FLAD, a flatness-decomposition framework that separates sharpness perturbations into gradient-aligned and stochastic-noise components, retaining only the noise part to promote generalization, plus a lightweight scheduling scheme. They formalize loss-landscape metrics with $R^{0}_{\rho}(w)$ and $R^{1}_{\rho}(w)$, present a decomposed optimization using EMA gradient approximations and Hessian-vector products, and prove convergence for nonconvex CL with a rate of $\mathcal{O}(\log n^T/\sqrt{n^T})$. Empirically, FLAD improves performance across diverse CL benchmarks when plugged into replay-, regularization-, and expansion-based methods, and even partial usage yields substantial efficiency gains, making curvature-guided CL practical at scale.

Abstract

Continual Learning (CL) aims to enable models to sequentially learn multiple tasks without forgetting previous knowledge. Recent studies have shown that optimizing towards flatter loss minima can improve model generalization. However, existing sharpness-aware methods for CL suffer from two key limitations: (1) they treat sharpness regularization as a unified signal without distinguishing the contributions of its components. and (2) they introduce substantial computational overhead that impedes practical deployment. To address these challenges, we propose FLAD, a novel optimization framework that decomposes sharpness-aware perturbations into gradient-aligned and stochastic-noise components, and show that retaining only the noise component promotes generalization. We further introduce a lightweight scheduling scheme that enables FLAD to maintain significant performance gains even under constrained training time. FLAD can be seamlessly integrated into various CL paradigms and consistently outperforms standard and sharpness-aware optimizers in diverse experimental settings, demonstrating its effectiveness and practicality in CL.

Beyond Sharpness: A Flatness Decomposition Framework for Efficient Continual Learning

TL;DR

Continual learning suffers from catastrophic forgetting, and while flatness of minima can improve generalization, prior sharpness-aware methods are costly and treat sharpness as a single signal. The authors propose FLAD, a flatness-decomposition framework that separates sharpness perturbations into gradient-aligned and stochastic-noise components, retaining only the noise part to promote generalization, plus a lightweight scheduling scheme. They formalize loss-landscape metrics with and , present a decomposed optimization using EMA gradient approximations and Hessian-vector products, and prove convergence for nonconvex CL with a rate of . Empirically, FLAD improves performance across diverse CL benchmarks when plugged into replay-, regularization-, and expansion-based methods, and even partial usage yields substantial efficiency gains, making curvature-guided CL practical at scale.

Abstract

Continual Learning (CL) aims to enable models to sequentially learn multiple tasks without forgetting previous knowledge. Recent studies have shown that optimizing towards flatter loss minima can improve model generalization. However, existing sharpness-aware methods for CL suffer from two key limitations: (1) they treat sharpness regularization as a unified signal without distinguishing the contributions of its components. and (2) they introduce substantial computational overhead that impedes practical deployment. To address these challenges, we propose FLAD, a novel optimization framework that decomposes sharpness-aware perturbations into gradient-aligned and stochastic-noise components, and show that retaining only the noise component promotes generalization. We further introduce a lightweight scheduling scheme that enables FLAD to maintain significant performance gains even under constrained training time. FLAD can be seamlessly integrated into various CL paradigms and consistently outperforms standard and sharpness-aware optimizers in diverse experimental settings, demonstrating its effectiveness and practicality in CL.
Paper Structure (23 sections, 1 theorem, 14 equations, 5 figures, 1 table, 1 algorithm)

This paper contains 23 sections, 1 theorem, 14 equations, 5 figures, 1 table, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Works
  3. Continual learning.
  4. Sharpness-aware minimization.
  5. Method
  6. Loss landscape sharpness.
  7. Optimization with sharpness.
  8. Continual learning.
  9. Empirical Analysis
  10. Effect of adversarial perturbation.
  11. Analysis of noise components.
  12. Convergence Analysis
  13. Experiments
  14. Experimental Setup
  15. Datasets.
  16. ...and 8 more sections

Key Result

Theorem 1

Assume the loss function is twice differentiable, bounded by $M$, and obeys the triangle inequality. Both the loss function and its second-order gradient are $\beta$-Lipschitz smooth. FLAD converges in all tasks with learning rate $\eta\leq 1/\beta$, the perturbation radius $\rho\leq 1/4\beta$, and where $n^T$ is the total iteration numbers of task $T$, $C_1=32M(\beta-1),C_2=32\gamma^2$ only depe

Figures (5)

  • Figure 1: Empirical analysis of perturbation components in zeroth- and first-order sharpness minimization. (a) Average accuracy across four training settings using different perturbation directions. (b) $\text{Tr}(\mathbf{H\Sigma})$ during training for stochastic-noise and original variants. (c) 2D loss landscape visualizations around final model parameters and training trajectory.
  • Figure 2: Analysis of generalization. In (a)(b)(c), we report Hessian eigenvalue distributions and the trace of SGD, C-Flat, and FLAD on replay N=5. In (d), we show the loss landscape around final model parameters on MEMO N=10.
  • Figure 3: Ablation experiments. In (a)(b), we conduct ablation experiments on parameter $\rho$ and $\gamma$. In (c)(d), we investigate the impact of our decomposition strategy across 6 CL methods.
  • Figure 4: To further reduce overhead, we apply FLAD in a limited number of epochs within each task on different method and different settings, the horizontal axis is the transition point, before and after which we use different optimizers (SGD or FLAD).
  • Figure 5: Convergence and computation overhead. In (a), we compare test accuracy of different optimizers during training process. In (b), we compare accuracy and training time of different optimizers.

Theorems & Definitions (1)

  • Theorem 1