CURLoRA: Stable LLM Continual Fine-Tuning and Catastrophic Forgetting Mitigation

TL;DR

CURLoRA tackles the problem of catastrophically forgetting adapted knowledge during continual fine-tuning of large language models while reducing trainable parameters. It introduces a CUR-decomposition based method that samples low-leverage columns and rows using inverted probabilities, initializes the central U matrix to zero, and keeps C and R fixed, so only U is updated. Theoretical analysis shows constrained parameter space, implicit regularization, reduced interference, and bounded output shift, while empirical results demonstrate superior forgetting mitigation and preserved perplexity across tasks, with substantial memory savings compared to full fine-tuning and standard LoRA. The approach yields stable task accuracy and robust general language modeling performance, highlighting its potential for resource-constrained continual learning scenarios and motivating future work on scalability, automatic hyperparameter selection, and integration with other adaptation techniques.

Abstract

This paper introduces CURLoRA, a novel approach to fine-tuning large language models (LLMs) that leverages CUR matrix decomposition in the context of Low-Rank Adaptation (LoRA). Our method addresses two critical challenges in LLM fine-tuning: mitigating catastrophic forgetting during continual learning and reducing the number of trainable parameters. We propose a unique modification to the CUR decomposition process, utilizing inverted probabilities for column and row selection which acts as an implicit regularization, and initializing the $U$ matrix as a zero matrix, and only fine-tuning it. We demonstrate through experiments on multiple datasets that CURLoRA outperforms standard LoRA in mitigating catastrophic forgetting. It maintains model stability and performance across tasks while significantly reducing the number of trainable parameters. Our results show that CURLoRA achieves very good and stable task accuracy while maintaining base model's perplexity scores fixed compared to LoRA upon continual fine-tuning, particularly in scenarios with limited data.