SLIM: Let LLM Learn More and Forget Less with Soft LoRA and Identity Mixture

TL;DR

This work tackles the costly and forgetful process of fine-tuning large language models by introducing SLIM, a mixture of soft LoRA adapters and identity layers with dynamic routing. SLIM uses weight yielding guided by sliding clustering to route inputs to either LoRA adapters or identity “highways,” and employs a fast dynamic merging technique to fuse LoRA components back into the base model, mitigating catastrophic forgetting without data replay. Empirical results on an OpenChat-8B base show SLIM achieving competitive downstream PEFT performance while significantly reducing forgetting on general tasks like MMLU, GSM8K, and PIQA. The approach offers a practical, parameter-efficient path to domain adaptation that preserves broad model capabilities while enabling task-focused learning.

Abstract

Although many efforts have been made, it is still a challenge to balance the training budget, downstream performance, and the general capabilities of the LLMs in many applications. Training the whole model for downstream tasks is expensive, and could easily result in catastrophic forgetting. By introducing parameter-efficient fine-tuning (PEFT), the training cost could be reduced, but it still suffers from forgetting, and limits the learning on the downstream tasks. To efficiently fine-tune the LLMs with less limitation to their downstream performance while mitigating the forgetting of general capabilities, we propose a novel mixture of expert (MoE) framework based on Soft LoRA and Identity Mixture (SLIM), that allows dynamic routing between LoRA adapters and skipping connection, enables the suppression of forgetting. We adopt weight-yielding with sliding clustering for better out-of-domain distinguish to enhance the routing. We also propose to convert the mixture of low-rank adapters to the model merging formulation and introduce fast dynamic merging of LoRA adapters to keep the general capabilities of the base model. Extensive experiments demonstrate that the proposed SLIM is comparable to the state-of-the-art PEFT approaches on the downstream tasks while achieving the leading performance in mitigating catastrophic forgetting.

Figures (6)

• Figure 1: The radar chart of the performance of the parameter-efficient fine-tuning methods. The blue benchmarks are evaluated after fine-tuning on other downstream datasets. Compared with the state-of-the-art parameter-efficient methods, the proposed SLIM could achieve comparable downstream performance while mitigating catastrophic forgetting.
• Figure 2: The overall framework of the proposed approach. "Iden" denotes the identity layer, which could be activated by the router as an information "highway" to bypass the LoRA adapters (see Sec. \ref{['sec:3.2']} for details), thereby preserving general capabilities. First, the router predicts the routing weight for each expert. Second, we calculate the distance from the input to the clusters and yield the routing weight accordingly. Then we activate the top-$K$ experts according to the yielded routing weight and fuse their output via dynamic merging. We remove the connections from the input to the weight matrices ($W$, LoRA adapters) and the "Iden" in the figure for simplicity.
• Figure 3: The FAST implementation of dynamic merging. By masking the sub-matrices of the low-rank adapter, we approximate the random sampling without introducing extra computational cost.
• Figure 4: The influence of masking ratio for the rows of $B$ and $A^T$ of LoRA adapters. The masking ratio is a trade-off between the downstream performance and the general capability.
• Figure 5: Model evaluation on the MMLU benchmark during the fine-tuning with LoRA, MixLoRA and the proposed SLIM on the MDS.