InsCL: A Data-efficient Continual Learning Paradigm for Fine-tuning Large Language Models with Instructions

TL;DR

This work proposes a novel paradigm called Instruction-based Continual Learning (InsCL), which dynamically replays previous data based on task similarity, calculated by Wasserstein Distance with instructions, and introduces an Instruction Information Metric (InsInfo) to quantify the complexity and diversity of instructions.

Abstract

Instruction tuning effectively optimizes Large Language Models (LLMs) for downstream tasks. Due to the changing environment in real-life applications, LLMs necessitate continual task-specific adaptation without catastrophic forgetting. Considering the heavy computational cost, replay-based Continual Learning (CL) methods are the simplest and most widely used for LLMs to address the forgetting issue. However, traditional replay-based methods do not fully utilize instructions to customize the replay strategy. In this work, we propose a novel paradigm called Instruction-based Continual Learning (InsCL). InsCL dynamically replays previous data based on task similarity, calculated by Wasserstein Distance with instructions. Moreover, we further introduce an Instruction Information Metric (InsInfo) to quantify the complexity and diversity of instructions. According to InsInfo, InsCL guides the replay process more inclined to high-quality data. We conduct extensive experiments over 16 tasks with different training orders, observing consistent performance improvements of InsCL. When all tasks have been trained, InsCL achieves performance gains of 3.0 Relative Gain compared with Random Replay, and 27.96 Relative Gain compared with No Replay.

Figures (5)

• Figure 1: The framework of InsCL, the index denotes task id. $D$ represents task data, and $R$ represents the sampled data to replay. InsCL dynamically replays $\alpha^*$ data for each previous task based on the task similarity calculated via Wasserstein Distance $W$. The dots represent instructions included in each task, and the darker colors represent higher InsInfo. The size of each color bar denotes the corresponding amount of replay data.
• Figure 2: We obtain 16 categories by integrating English tasks in the SuperNI dataset. And we conduct further experiments based on 16 reallocated tasks.
• Figure 3: Progressive Relative Gain results for LLaMA-7B in continual instruction tuning. We set Relative Gain to 100 for training on the first task, denoting the initial performance without forgetting. When it comes to stage $i$, we plot the average score of corresponding Relative Gain with three different training orders. The closer the Relative Gain is to 100, the better to alleviate catastrophic forgetting and preserve knowledge.
• Figure 4: We analyze the forgetting rate based on Curriculum training order. The results of all previous tasks are reported when training is finished on the last task.
• Figure 5: The analysis of forgetting category. We divide forgetting instances into Instruction-Related and Instruction Unrelated. After training on Curriculum order, the ratios of two categories in previous tasks are reported.