Recent Advances of Foundation Language Models-based Continual Learning: A Survey

TL;DR

This survey maps how foundation language models—PLMs, LLMs, and VLMs—can continually acquire new knowledge without catastrophically forgetting, by organizing approaches into offline vs online CL and across four method families: traditional CL, continual pre-training, parameter-efficient tuning, and instruction tuning. It provides a comprehensive taxonomy by model type and CL setting, surveys representative methods for domain-, task-, and class-incremental learning, and catalogs datasets and metrics used to evaluate forgetting and transfer. The paper highlights current challenges (e.g., benchmarks, privacy, multi-modal integration) and outlines a roadmap toward autonomous, cognitively-informed, and robust lifelong learning for NLP and multimodal systems. Collectively, these insights advance understanding of how to deploy scalable, efficient, and adaptable foundation LMs in dynamic real-world environments.

Abstract

Recently, foundation language models (LMs) have marked significant achievements in the domains of natural language processing (NLP) and computer vision (CV). Unlike traditional neural network models, foundation LMs obtain a great ability for transfer learning by acquiring rich commonsense knowledge through pre-training on extensive unsupervised datasets with a vast number of parameters. However, they still can not emulate human-like continuous learning due to catastrophic forgetting. Consequently, various continual learning (CL)-based methodologies have been developed to refine LMs, enabling them to adapt to new tasks without forgetting previous knowledge. However, a systematic taxonomy of existing approaches and a comparison of their performance are still lacking, which is the gap that our survey aims to fill. We delve into a comprehensive review, summarization, and classification of the existing literature on CL-based approaches applied to foundation language models, such as pre-trained language models (PLMs), large language models (LLMs) and vision-language models (VLMs). We divide these studies into offline CL and online CL, which consist of traditional methods, parameter-efficient-based methods, instruction tuning-based methods and continual pre-training methods. Offline CL encompasses domain-incremental learning, task-incremental learning, and class-incremental learning, while online CL is subdivided into hard task boundary and blurry task boundary settings. Additionally, we outline the typical datasets and metrics employed in CL research and provide a detailed analysis of the challenges and future work for LMs-based continual learning.

Figures (9)

• Figure 1: Comparison between traditional CL and Foundation language models (LMs)-Based CL.
• Figure 2: Taxonomy of foundation language models for continual learning.
• Figure 3: The setting of different offline continual learning tasks, including task-incremental learning, class-incremental learning and domain-incremental learning. The samples with different classes (domains) are marked with various shapes (colors).
• Figure 4: The setting of different online continual learning tasks, including hard task boundary arriving and blurry task boundary arriving. The samples with different classes (domains) are marked with various shapes (colors).
• Figure 5: Frameworks in DIL: PlugLM (PLM-based) cheng2022language, Lifelong-MoE (LLM-based) chen2023lifelong, S-Prompts (VLM-based) wang2022s.