Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

MePo: Meta Post-Refinement for Rehearsal-Free General Continual Learnin

Guanglong Sun, Hongwei Yan, Liyuan Wang, Zhiqi Kang, Shuang Cui, Hang Su, Jun Zhu, Yi Zhong

TL;DR

This paper tackles the challenge of general continual learning (GCL) under rehearsal-free constraints by introducing Meta Post-Refinement (MePo). MePo constructs pseudo task sequences from pretraining data and employs a bi-level meta-learning framework to refine the pretrained backbone, yielding a CL-tailored initialization that adapts rapidly to downstream GCL tasks. Additionally, MePo initializes a meta covariance matrix to align second-order statistics of feature representations, stabilizing output predictions through covariance-based alignment. Empirical results across CIFAR-100, ImageNet-R, and CUB-200 show substantial improvements over strong PTMs-based baselines with minimal overhead, especially when using self-supervised pretrained checkpoints, demonstrating the method’s effectiveness and efficiency in rehearsal-free GCL settings.

Abstract

To cope with uncertain changes of the external world, intelligent systems must continually learn from complex, evolving environments and respond in real time. This ability, collectively known as general continual learning (GCL), encapsulates practical challenges such as online datastreams and blurry task boundaries. Although leveraging pretrained models (PTMs) has greatly advanced conventional continual learning (CL), these methods remain limited in reconciling the diverse and temporally mixed information along a single pass, resulting in sub-optimal GCL performance. Inspired by meta-plasticity and reconstructive memory in neuroscience, we introduce here an innovative approach named Meta Post-Refinement (MePo) for PTMs-based GCL. This approach constructs pseudo task sequences from pretraining data and develops a bi-level meta-learning paradigm to refine the pretrained backbone, which serves as a prolonged pretraining phase but greatly facilitates rapid adaptation of representation learning to downstream GCL tasks. MePo further initializes a meta covariance matrix as the reference geometry of pretrained representation space, enabling GCL to exploit second-order statistics for robust output alignment. MePo serves as a plug-in strategy that achieves significant performance gains across a variety of GCL benchmarks and pretrained checkpoints in a rehearsal-free manner (e.g., 15.10\%, 13.36\%, and 12.56\% on CIFAR-100, ImageNet-R, and CUB-200 under Sup-21/1K). Our source code is available at \href{https://github.com/SunGL001/MePo}{MePo}

MePo: Meta Post-Refinement for Rehearsal-Free General Continual Learnin

TL;DR

This paper tackles the challenge of general continual learning (GCL) under rehearsal-free constraints by introducing Meta Post-Refinement (MePo). MePo constructs pseudo task sequences from pretraining data and employs a bi-level meta-learning framework to refine the pretrained backbone, yielding a CL-tailored initialization that adapts rapidly to downstream GCL tasks. Additionally, MePo initializes a meta covariance matrix to align second-order statistics of feature representations, stabilizing output predictions through covariance-based alignment. Empirical results across CIFAR-100, ImageNet-R, and CUB-200 show substantial improvements over strong PTMs-based baselines with minimal overhead, especially when using self-supervised pretrained checkpoints, demonstrating the method’s effectiveness and efficiency in rehearsal-free GCL settings.

Abstract

To cope with uncertain changes of the external world, intelligent systems must continually learn from complex, evolving environments and respond in real time. This ability, collectively known as general continual learning (GCL), encapsulates practical challenges such as online datastreams and blurry task boundaries. Although leveraging pretrained models (PTMs) has greatly advanced conventional continual learning (CL), these methods remain limited in reconciling the diverse and temporally mixed information along a single pass, resulting in sub-optimal GCL performance. Inspired by meta-plasticity and reconstructive memory in neuroscience, we introduce here an innovative approach named Meta Post-Refinement (MePo) for PTMs-based GCL. This approach constructs pseudo task sequences from pretraining data and develops a bi-level meta-learning paradigm to refine the pretrained backbone, which serves as a prolonged pretraining phase but greatly facilitates rapid adaptation of representation learning to downstream GCL tasks. MePo further initializes a meta covariance matrix as the reference geometry of pretrained representation space, enabling GCL to exploit second-order statistics for robust output alignment. MePo serves as a plug-in strategy that achieves significant performance gains across a variety of GCL benchmarks and pretrained checkpoints in a rehearsal-free manner (e.g., 15.10\%, 13.36\%, and 12.56\% on CIFAR-100, ImageNet-R, and CUB-200 under Sup-21/1K). Our source code is available at \href{https://github.com/SunGL001/MePo}{MePo}
Paper Structure (21 sections, 1 theorem, 28 equations, 9 figures, 12 tables, 1 algorithm)

This paper contains 21 sections, 1 theorem, 28 equations, 9 figures, 12 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Formulation and Preliminary Analysis
  3. Problem Setup
  4. Empirical Analysis of PTMs-Based Methods
  5. Meta Post-Refinement
  6. MePo for Representation Learning
  7. MePo for Output Alignment
  8. Feature Alignment.
  9. Experiments
  10. Conclusion and Discussion
  11. Related Work
  12. Experimental Setup
  13. Additional Results
  14. Theoretical Analysis of Meta-Rep
  15. Setup and Notation
  16. ...and 6 more sections

Key Result

Theorem 4.1

Let $\theta^\star$ be a stationary point of the surrogate meta-objective: obtained by applying the meta-update in eq:meta_update. Assume each loss $L_t$ and $L_{\text{joint}}$ is twice differentiable, with bounded gradients and Hessians, and the inner-loop step size $\eta$ is sufficiently small. Then for any two-task sequence $A \rightarrow B$ and some constant $C > 0$, t Moreover, if the pseudo-

Figures (9)

  • Figure 1: The proposed MePo framework for rehearsal-free general continual learning.
  • Figure 2: Empirical analysis of PTMs-based methods under different experimental setups. We compare (a) Offline CL vs General CL, (b) Offline CL vs Online CL, and (c) Online CL vs General CL. "-Rep", without logit masking. "-Out", without representation learning.
  • Figure 3: MePo representation learning.
  • Figure 4: MePo feature alignment.
  • Figure 5: Empirical evaluation of the combination weight $\alpha$ in MePo. Here we employ $A_{{\rm{AUC}}} (\uparrow)$ as the evaluation metric. The complete quantification results are included in Appendix Tables \ref{['table.hyper_auc']} and \ref{['table.hyper_last']}.
  • ...and 4 more figures

Theorems & Definitions (1)

  • Theorem 4.1: Sequential Update Consistency Theorem