Refine Large Language Model Fine-tuning via Instruction Vector

TL;DR

This work tackles catastrophic forgetting during instruction fine-tuning of large language models by distinguishing world knowledge from instruction-following using $P(y|x)$ and $P(y^c|c,x)$. It introduces the Instruction Vector (IV), a task-conditioned latent representation built from selected attention-head activations that governs instruction-following performance via a computation graph $f_M(x,c,\theta_c)\rightarrow y_c$, and demonstrates that forgetting arises from overlay of new patterns on the IV pathways rather than erasing prior skills. The authors propose IV-guided training with progressive IV intervention and an IV-based KL loss to preserve the original computation graph, achieving substantial reductions in forgetting and preserved plasticity across TRACE, FUNC, and LONG benchmarks. This provides mechanistic insight into how instruction-following capabilities evolve during fine-tuning and offers a practical method to maintain pre-existing capabilities in continual instruction learning, with potential for extension to additional backbones and tasks.

Abstract

Fine-tuning large language models (LLMs) can cause them to lose their general capabilities. However, the intrinsic mechanisms behind such forgetting remain unexplored. In this paper, we begin by examining this phenomenon by focusing on knowledge understanding and instruction following, with the latter identified as the main contributor to forgetting during fine-tuning. Consequently, we propose the Instruction Vector (IV) framework to capture model representations highly related to specific instruction-following capabilities, thereby making it possible to understand model-intrinsic forgetting. Through the analysis of IV dynamics pre and post-training, we suggest that fine-tuning mostly adds specialized reasoning patterns instead of erasing previous skills, which may appear as forgetting. Building on this insight, we develop IV-guided training, which aims to preserve original computation graph, thereby mitigating catastrophic forgetting. Empirical tests on three benchmarks confirm the efficacy of this new approach, supporting the relationship between IVs and forgetting. Our code will be made available soon.

Figures (6)

• Figure 1: Instruction vector hypothesis for LLM understanding. $\theta_c$ is extracted by aggregating representations of attention heads identified to have causal influence to the output. Forgetting is resulted from the suppression of instruction vector associated computation graph.
• Figure 2: Task in world knowledge form $(x,y)$ and instruction form $(x,c,y^c)$.
• Figure 3: Accuracy curve across naive sequential instruction fine-tuning on the TRACE benchmark. X-axis delineates the stages through training, with "M0" indicating the original pre-trained model, and "Mi" signifying the model post-instruction fine-tuning for the i-th task in sequence. The tasks follow the sequence of Cstance, Fomc, Meetingbank, Py150, ScienceQA, and Numgluecm. Y-axis indicates the rank classification accuracy. Notably, the first four datasets are absent from the training set, whereas the final three datasets are part of the training distribution.
• Figure 4: Illustration of the instruction vector hypothesis. Here, $x$ represents the context, $c$ stands for a specific instruction, $y_c$ is the desirable output, and $\theta_c$ denotes the instruction vector. From (a) to (g), it visually details how these variables interact under different model conditions, with the accuracy above correlating to the respective performance on the CommonsenseQA task. The model configuration depicted in (d) is identified as the best state.
• Figure 5: Intervention results on four datasets via Enhanced Instruction Vector.