Mixtures of SubExperts for Large Language Continual Learning

TL;DR

A novel adaptive PEFT method referred to as Mixtures of SubExperts (MoSEs) is proposed, a novel continual learning framework designed for minimal forgetting and efficient scalability, which significantly outperform conventional continual learning approaches in both knowledge retention and scalability to new tasks.

Abstract

Adapting Large Language Models (LLMs) to a continuous stream of tasks is a critical yet challenging endeavor. While Parameter-Efficient Fine-Tuning (PEFT) methods have become a standard for this, they face a fundamental dilemma in continual learning. Reusing a single set of PEFT parameters for new tasks often leads to catastrophic forgetting of prior knowledge. Conversely, allocating distinct parameters for each task prevents forgetting but results in a linear growth of the model's size and fails to facilitate knowledge transfer between related tasks. To overcome these limitations, we propose a novel adaptive PEFT method referred to as \textit{Mixtures of SubExperts (MoSEs)}, a novel continual learning framework designed for minimal forgetting and efficient scalability. MoSEs integrate a sparse Mixture of SubExperts into the transformer layers, governed by a task-specific routing mechanism. This architecture allows the model to isolate and protect knowledge within dedicated SubExperts, thereby minimizing parameter interference and catastrophic forgetting. Crucially, the router can adaptively select and combine previously learned sparse parameters for new tasks, enabling effective knowledge transfer while ensuring that the model's capacity grows sublinearly. We evaluate MoSEs on the comprehensive TRACE benchmark datasets. Our experiments demonstrate that MoSEs significantly outperform conventional continual learning approaches in both knowledge retention and scalability to new tasks, achieving state-of-the-art performance with substantial memory and computational savings.

Figures (5)

• Figure 1: Continual LLMs: the objective is to design a fine-tuned transformer that works well across a sequential task: two types of parameter-efficient fine-tuning (PEFT) methods through Low-Rank Adaptation (LoRA) at attention layers and Mixture of Experts (MoEs) at feed forward network (FFN) layer. $\bm{A}, \bm{B}$ are the learnable parameters, initialized respectively.
• Figure 2: Mixtures of SubExperts (MoSEs): The Self-Attention layer is fine-tuned by MoSEs to operate on task-specific tokens. Given, $\bm{x}_i^{t-1}$ and $\bm{x}_i^{t}$, the MoSEs router adaptively distributes them across $N$ sub-experts. Each token is routed to the most relevant subexperts, and the final output is computed as the weighted sum of the selected subexpert outputs, where the weights correspond to the router gate values (e.g., $p = 0.65$ or $p = 0.72$).
• Figure 3: Average Performances and Model Capasity of MoSEs on TRACE 0.5K.
• Figure 4: (TaIL) Transfer Matrixes on the TRACE (0.5K) measured by source and target.
• Figure 5: Comparisions of Transfer Matrixes on the TRACE (0.5K). TIL denotes that task ID is given in inference.