Learn or Recall? Revisiting Incremental Learning with Pre-trained Language Models

TL;DR

A frustratingly easy method called SEQ* for IL with PLMs is proposed that has competitive or superior performance compared to state-of-the-art (SOTA) IL methods and requires considerably less trainable parameters and training time.

Abstract

Incremental Learning (IL) has been a long-standing problem in both vision and Natural Language Processing (NLP) communities. In recent years, as Pre-trained Language Models (PLMs) have achieved remarkable progress in various NLP downstream tasks, utilizing PLMs as backbones has become a common practice in recent research of IL in NLP. Most assume that catastrophic forgetting is the biggest obstacle to achieving superior IL performance and propose various techniques to overcome this issue. However, we find that this assumption is problematic. Specifically, we revisit more than 20 methods on four classification tasks (Text Classification, Intent Classification, Relation Extraction, and Named Entity Recognition) under the two most popular IL settings (Class-Incremental and Task-Incremental) and reveal that most of them severely underestimate the inherent anti-forgetting ability of PLMs. Based on the observation, we propose a frustratingly easy method called SEQ* for IL with PLMs. The results show that SEQ* has competitive or superior performance compared to state-of-the-art (SOTA) IL methods and requires considerably less trainable parameters and training time. These findings urge us to revisit the IL with PLMs and encourage future studies to have a fundamental understanding of the catastrophic forgetting in PLMs. The data, code and scripts are publicly available at https://github.com/zzz47zzz/codebase-for-incremental-learning-with-llm.

Figures (27)

• Figure 1: The comparison between the proposed SEQ* and SOTA IL methods on five class-incremental tasks. We report the average accuracy after learning the final task. The detailed results are provided in Table \ref{['tab:sota_main_gen_pythia410m']}.
• Figure 2: An illustration of how we obtain the probing and the observed performance of the model when learning the fourth task if there are a total of 15 tasks. The observed performance is used as a metric of forgetting in existing studies. The probing performance indicates how the encoder forgets. However, it is overlooked by previous studies.
• Figure 3: The observed and probing performance on Class-Incremental Intent Classification. The dataset is CLINC150. The backbones are generative models. (a)(b) are the observed performance and training loss during IL training. (c)-(f) are the probing performance when different metrics are adopted.
• Figure 4: The histogram of features and different embeddings of Pythia-410m. The features are calculated on the training set of CLINC150, and the output word embeddings are loaded from pre-trained weights. The class embeddings refer to the row vectors of the weight matrix in the probing classifier on CLINC150. The class prototypes refer to the class feature centres estimated on the training set of CLINC150.
• Figure 5: The linear probing performance on checkpoints with different pre-training steps. (a) and (b) are evaluated before and after incremental learning using SEQ. "Intent" and "RE" represent the model is evaluated on the Class-Incremental Intent Classification or Relation Extraction.