The Effectiveness of Approximate Regularized Replay for Efficient Supervised Fine-Tuning of Large Language Models

TL;DR

The paper addresses catastrophic forgetting during supervised fine-tuning of large language models using parameter-efficient LoRA adapters. It introduces approximate regularized replay, combining KL divergence regularization with interleaved open-web data to stabilize learning while preserving plasticity, and validates the approach on Qwen instruction-tuned models across multiple tasks and model sizes. Theoretical justifications connect KL regularization to Bayesian priors and information bottleneck principles, while replay is framed as steering optimization toward the future data distribution. Empirically, standard LoRA fine-tuning suffers substantial forgetting, which is mitigated by replay and KL regularization, with the best results arising from a judicious combination that achieves near-zero forgetting and preserved or enhanced plasticity, all at modest computational overhead. The work offers a practical, open-data–based strategy to democratize efficient, robust instruction-tuning of LLMs for real-world applications.

Abstract

Although parameter-efficient fine-tuning methods, such as LoRA, only modify a small subset of parameters, they can have a significant impact on the model. Our instruction-tuning experiments show that LoRA-based supervised fine-tuning can catastrophically degrade model capabilities, even when trained on very small datasets for relatively few steps. With that said, we demonstrate that while the most straightforward approach (that is likely the most used in practice) fails spectacularly, small tweaks to the training procedure with very little overhead can virtually eliminate the problem. Particularly, in this paper we consider a regularized approximate replay approach which penalizes KL divergence with respect to the initial model and interleaves in data for next token prediction from a different, yet similar, open access corpus to what was used in pre-training. When applied to Qwen instruction-tuned models, we find that this recipe preserves general knowledge in the model without hindering plasticity to new tasks by adding a modest amount of computational overhead.